From patchwork Thu Sep 8 12:55:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6ZmI5piK?= X-Patchwork-Id: 37764 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:139a:b0:8f:1db5:eae2 with SMTP id w26csp268096pzh; Thu, 8 Sep 2022 05:56:17 -0700 (PDT) X-Google-Smtp-Source: AA6agR7oOtuYYPXPX5IggHgsBm3Bx6ZMe+8JQNuXkTNQhOal3x1Ged5Zkpm/plcUAgemXh7Mf4To X-Received: by 2002:a17:907:a06f:b0:770:5132:a3a5 with SMTP id ia15-20020a170907a06f00b007705132a3a5mr6055883ejc.69.1662641777414; Thu, 08 Sep 2022 05:56:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1662641777; cv=none; d=google.com; s=arc-20160816; b=s2ibVdO+t8TWkeLMrww0DWt43EzwNWQdV5SJ5VqUYDT3U1OcRLIkyzj6baqezcvkXR LUkGlnbZ0+e7FStvM04mF0MmHIXO5v2YAzGCXSusA15RYTXbRGe7ISyRgUZAZQ8CATaX /9GTivrdno/HTWGZN7hqjqODMD17vfZfGuXqooC+jYCyiVZAYceMLHy5aHH9RcZdEQYd CgqdD2cZXmqbn5qkA2NjPMKzrsPsqIpiOwq+VH/b/emJ6VP4RGmuHsOjqOktc7kfFGe3 9oX24FJH8FHRuxx5mRWEZJda6XkEtpbFP68zTO7WJDHAQa0otHDISuzB6mrukPQ29BQz Dn5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=8ejX8uurDmMNAnC+8mdc2wQwGiMZS/sgITbZtHO9Wi8=; b=DERkd4SBhPzdVjxgRuID20JgjLHcvOj/mvJsScaM5Zui15CkK3ug16YUmo5D4InLWy XqKfTmf9+t2nHTMottoRlml4XbNYGpbMmeS7Cd696/+hyi2Og3kENkr467gjwcBJ7bvk c63Prg1jLwmYoJ+wjj5goSySRz5no3sYxFcs8SN84bvR8PVAEJ0mQ9LMNIBfEyni4whO JkLoRvsFJNpIOiLMcqBrbl7teNJN38GzgrtwtxnCagqY9hRYLty5SXVv2t21nazB4cTw /liijJd8kuIFd58rfReFs4C6fOttfYnbW0TNSXqY12rU12Hf9k/rQ6CkKmABHgljWdjc TyfQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z18-20020a170906815200b00772919caedbsi1484857ejw.581.2022.09.08.05.56.16; Thu, 08 Sep 2022 05:56:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 769F568B9CB; Thu, 8 Sep 2022 15:56:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from loongson.cn (mail.loongson.cn [114.242.206.163]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 56FF168B9CB for ; Thu, 8 Sep 2022 15:56:06 +0300 (EEST) Received: from localhost (unknown [36.33.26.144]) by localhost.localdomain (Coremail) with SMTP id AQAAf8DxPGs55hljnHEUAA--.21589S3; Thu, 08 Sep 2022 20:55:21 +0800 (CST) From: Hao Chen To: ffmpeg-devel@ffmpeg.org Date: Thu, 8 Sep 2022 20:55:18 +0800 Message-Id: <20220908125519.15807-2-chenhao@loongson.cn> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220908125519.15807-1-chenhao@loongson.cn> References: <20220908125519.15807-1-chenhao@loongson.cn> MIME-Version: 1.0 X-CM-TRANSID: AQAAf8DxPGs55hljnHEUAA--.21589S3 X-Coremail-Antispam: 1UD129KBjvAXoW3KrW7ZrWrurW7Ar1rKF45ZFb_yoW8Gr1rto WjqFWkA34kGrZ7C3y3Ar18GasrXF47Wr1kZa1rtr1UXa4F9345A343Zw4SqayDKr4Fg345 Gr93GrWxJFsxAr98n29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7v73VFW2AGmfu7bjvjm3 AaLaJ3UjIYCTnIWjp_UUUYj7k0a2IF6w4kM7kC6x804xWl14x267AKxVWUJVW8JwAFc2x0 x2IEx4CE42xK8VAvwI8IcIk0rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2ocxC64kIII0Yj4 1l84x0c7CEw4AK67xGY2AK021l84ACjcxK6xIIjxv20xvE14v26ryj6F1UM28EF7xvwVC0 I7IYx2IY6xkF7I0E14v26F4j6r4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwV C2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC 0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUAVWUtwAv7VC2z280aVAFwI0_Gr0_Cr 1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JMxkIecxEwVAFwVW5JwCF04k2 0xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI 8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jrv_JF1lIxkGc2Ij64vIr41l IxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr0_Gr1lIx AIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2 jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5izV5UUUUU== X-CM-SenderInfo: hfkh0xtdr6z05rqj20fqof0/ Subject: [FFmpeg-devel] [PATCH v1 1/2] Fix bugs in me_cmp_msa.c file. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Lu Wang Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: yksbqAQDI/ZP From: Lu Wang This patch fixes a bug where the fate-checkasm-motion fails when h is not a multiple of 8. --- libavcodec/mips/me_cmp_msa.c | 201 ++++++++++++++++++++++++++++++----- 1 file changed, 173 insertions(+), 28 deletions(-) diff --git a/libavcodec/mips/me_cmp_msa.c b/libavcodec/mips/me_cmp_msa.c index 00a3cfd53f..351494161f 100644 --- a/libavcodec/mips/me_cmp_msa.c +++ b/libavcodec/mips/me_cmp_msa.c @@ -25,11 +25,13 @@ static uint32_t sad_8width_msa(const uint8_t *src, int32_t src_stride, const uint8_t *ref, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 2; + int res = (height & 0x03); v16u8 src0, src1, src2, src3, ref0, ref1, ref2, ref3; + v8u16 zero = { 0 }; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 2); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); LD_UB4(ref, ref_stride, ref0, ref1, ref2, ref3); @@ -39,6 +41,16 @@ static uint32_t sad_8width_msa(const uint8_t *src, int32_t src_stride, src0, src1, ref0, ref1); sad += SAD_UB2_UH(src0, src1, ref0, ref1); } + for (; res--; ) { + v16u8 diff; + src0 = LD_UB(src); + ref0 = LD_UB(ref); + src += src_stride; + ref += ref_stride; + diff = __msa_asub_u_b((v16u8) src0, (v16u8) ref0); + diff = (v16u8)__msa_ilvr_d((v2i64)zero, (v2i64)diff); + sad += __msa_hadd_u_h((v16u8) diff, (v16u8) diff); + } return (HADD_UH_U32(sad)); } @@ -47,11 +59,12 @@ static uint32_t sad_16width_msa(const uint8_t *src, int32_t src_stride, const uint8_t *ref, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 2; + int res = (height & 0x03); v16u8 src0, src1, ref0, ref1; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 2); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB2(src, src_stride, src0, src1); src += (2 * src_stride); LD_UB2(ref, ref_stride, ref0, ref1); @@ -64,7 +77,15 @@ static uint32_t sad_16width_msa(const uint8_t *src, int32_t src_stride, ref += (2 * ref_stride); sad += SAD_UB2_UH(src0, src1, ref0, ref1); } - + for (; res > 0; res--) { + v16u8 diff; + src0 = LD_UB(src); + ref0 = LD_UB(ref); + src += src_stride; + ref += ref_stride; + diff = __msa_asub_u_b((v16u8) src0, (v16u8) ref0); + sad += __msa_hadd_u_h((v16u8) diff, (v16u8) diff); + } return (HADD_UH_U32(sad)); } @@ -74,12 +95,14 @@ static uint32_t sad_horiz_bilinear_filter_8width_msa(const uint8_t *src, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 3; + int32_t res = height & 0x07; v16u8 src0, src1, src2, src3, comp0, comp1; v16u8 ref0, ref1, ref2, ref3, ref4, ref5; + v8u16 zero = { 0 }; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 3); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); LD_UB4(ref, ref_stride, ref0, ref1, ref2, ref3); @@ -107,6 +130,18 @@ static uint32_t sad_horiz_bilinear_filter_8width_msa(const uint8_t *src, sad += SAD_UB2_UH(src0, src1, comp0, comp1); } + for (; res--; ) { + v16u8 diff; + src0 = LD_UB(src); + ref0 = LD_UB(ref); + ref1 = LD_UB(ref + 1); + src += src_stride; + ref += ref_stride; + comp0 = (v16u8)__msa_aver_u_b((v16u8) ref0, (v16u8) ref1); + diff = __msa_asub_u_b((v16u8) src0, (v16u8) comp0); + diff = (v16u8)__msa_ilvr_d((v2i64) zero, (v2i64) diff); + sad += __msa_hadd_u_h((v16u8) diff, (v16u8) diff); + } return (HADD_UH_U32(sad)); } @@ -116,12 +151,13 @@ static uint32_t sad_horiz_bilinear_filter_16width_msa(const uint8_t *src, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 3; + int32_t res = height & 0x07; v16u8 src0, src1, src2, src3, comp0, comp1; v16u8 ref00, ref10, ref20, ref30, ref01, ref11, ref21, ref31; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 3); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); LD_UB4(ref, ref_stride, ref00, ref10, ref20, ref30); @@ -145,6 +181,17 @@ static uint32_t sad_horiz_bilinear_filter_16width_msa(const uint8_t *src, sad += SAD_UB2_UH(src2, src3, comp0, comp1); } + for (; res--; ) { + v16u8 diff; + src0 = LD_UB(src); + ref00 = LD_UB(ref); + ref01 = LD_UB(ref + 1); + src += src_stride; + ref += ref_stride; + comp0 = (v16u8)__msa_aver_u_b((v16u8) ref00, (v16u8) ref01); + diff = __msa_asub_u_b((v16u8) src0, (v16u8) comp0); + sad += __msa_hadd_u_h((v16u8) diff, (v16u8) diff); + } return (HADD_UH_U32(sad)); } @@ -154,12 +201,14 @@ static uint32_t sad_vert_bilinear_filter_8width_msa(const uint8_t *src, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 3; + int32_t res = height & 0x07; v16u8 src0, src1, src2, src3, comp0, comp1; v16u8 ref0, ref1, ref2, ref3, ref4; + v8u16 zero = { 0 }; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 3); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); LD_UB5(ref, ref_stride, ref0, ref1, ref2, ref3, ref4); @@ -183,6 +232,17 @@ static uint32_t sad_vert_bilinear_filter_8width_msa(const uint8_t *src, sad += SAD_UB2_UH(src0, src1, comp0, comp1); } + for (; res--; ) { + v16u8 diff; + src0 = LD_UB(src); + LD_UB2(ref, ref_stride, ref0, ref1); + src += src_stride; + ref += ref_stride; + comp0 = (v16u8)__msa_aver_u_b((v16u8) ref0, (v16u8) ref1); + diff = __msa_asub_u_b((v16u8) src0, (v16u8) comp0); + diff = (v16u8)__msa_ilvr_d((v2i64) zero, (v2i64) diff); + sad += __msa_hadd_u_h((v16u8) diff, (v16u8) diff); + } return (HADD_UH_U32(sad)); } @@ -192,12 +252,13 @@ static uint32_t sad_vert_bilinear_filter_16width_msa(const uint8_t *src, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 3; + int32_t res = height & 0x07; v16u8 src0, src1, src2, src3, comp0, comp1; v16u8 ref0, ref1, ref2, ref3, ref4; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 3); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB5(ref, ref_stride, ref4, ref0, ref1, ref2, ref3); ref += (5 * ref_stride); LD_UB4(src, src_stride, src0, src1, src2, src3); @@ -221,6 +282,16 @@ static uint32_t sad_vert_bilinear_filter_16width_msa(const uint8_t *src, sad += SAD_UB2_UH(src2, src3, comp0, comp1); } + for (; res--; ) { + v16u8 diff; + src0 = LD_UB(src); + LD_UB2(ref, ref_stride, ref0, ref1); + src += src_stride; + ref += ref_stride; + comp0 = (v16u8)__msa_aver_u_b((v16u8) ref0, (v16u8) ref1); + diff = __msa_asub_u_b((v16u8) src0, (v16u8) comp0); + sad += __msa_hadd_u_h((v16u8) diff, (v16u8) diff); + } return (HADD_UH_U32(sad)); } @@ -230,11 +301,13 @@ static uint32_t sad_hv_bilinear_filter_8width_msa(const uint8_t *src, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 2; + int32_t res = height & 0x03; v16u8 src0, src1, src2, src3, temp0, temp1, diff; v16u8 ref0, ref1, ref2, ref3, ref4; v16i8 mask = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; v8u16 comp0, comp1, comp2, comp3; + v8u16 zero = { 0 }; v8u16 sad = { 0 }; for (ht_cnt = (height >> 2); ht_cnt--;) { @@ -277,6 +350,22 @@ static uint32_t sad_hv_bilinear_filter_8width_msa(const uint8_t *src, sad += __msa_hadd_u_h(diff, diff); } + for (; res--; ) { + src0 = LD_UB(src); + LD_UB2(ref, ref_stride, ref0, ref1); + temp0 = (v16u8) __msa_vshf_b(mask, (v16i8) ref0, (v16i8) ref0); + temp1 = (v16u8) __msa_vshf_b(mask, (v16i8) ref1, (v16i8) ref1); + src += src_stride; + ref += ref_stride; + comp0 = __msa_hadd_u_h(temp0, temp0); + comp2 = __msa_hadd_u_h(temp1, temp1); + comp2 += comp0; + comp2 = (v8u16)__msa_srari_h((v8i16) comp2, 2); + comp0 = (v16u8) __msa_pckev_b((v16i8) zero, (v16i8) comp2); + diff = __msa_asub_u_b(src0, comp0); + diff = (v16u8)__msa_ilvr_d((v2i64) zero, (v2i64) diff); + sad += __msa_hadd_u_h(diff, diff); + } return (HADD_UH_U32(sad)); } @@ -286,14 +375,15 @@ static uint32_t sad_hv_bilinear_filter_16width_msa(const uint8_t *src, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 3; + int32_t res = height & 0x07; v16u8 src0, src1, src2, src3, comp, diff; v16u8 temp0, temp1, temp2, temp3; v16u8 ref00, ref01, ref02, ref03, ref04, ref10, ref11, ref12, ref13, ref14; v8u16 comp0, comp1, comp2, comp3; v8u16 sad = { 0 }; - for (ht_cnt = (height >> 3); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); LD_UB5(ref, ref_stride, ref04, ref00, ref01, ref02, ref03); @@ -389,6 +479,25 @@ static uint32_t sad_hv_bilinear_filter_16width_msa(const uint8_t *src, diff = __msa_asub_u_b(src3, comp); sad += __msa_hadd_u_h(diff, diff); } + for (; res--; ) { + src0 = LD_UB(src); + LD_UB2(ref, ref_stride, ref00, ref10); + LD_UB2(ref + 1, ref_stride, ref01, ref11); + src += src_stride; + ref += ref_stride; + ILVRL_B2_UB(ref10, ref00, temp0, temp1); + ILVRL_B2_UB(ref11, ref01, temp2, temp3); + comp0 = __msa_hadd_u_h(temp0, temp0); + comp1 = __msa_hadd_u_h(temp1, temp1); + comp2 = __msa_hadd_u_h(temp2, temp2); + comp3 = __msa_hadd_u_h(temp3, temp3); + comp2 += comp0; + comp3 += comp1; + SRARI_H2_UH(comp2, comp3, 2); + comp = (v16u8) __msa_pckev_b((v16i8) comp3, (v16i8) comp2); + diff = __msa_asub_u_b(src0, comp); + sad += __msa_hadd_u_h(diff, diff); + } return (HADD_UH_U32(sad)); } @@ -407,15 +516,17 @@ static uint32_t sse_4width_msa(const uint8_t *src_ptr, int32_t src_stride, const uint8_t *ref_ptr, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 2; + int32_t res = height & 0x03; uint32_t sse; uint32_t src0, src1, src2, src3; uint32_t ref0, ref1, ref2, ref3; - v16u8 src = { 0 }; - v16u8 ref = { 0 }; - v4i32 var = { 0 }; + v16u8 src = { 0 }; + v16u8 ref = { 0 }; + v16u8 zero = { 0 }; + v4i32 var = { 0 }; - for (ht_cnt = (height >> 2); ht_cnt--;) { + for (; ht_cnt--; ) { LW4(src_ptr, src_stride, src0, src1, src2, src3); src_ptr += (4 * src_stride); LW4(ref_ptr, ref_stride, ref0, ref1, ref2, ref3); @@ -426,6 +537,20 @@ static uint32_t sse_4width_msa(const uint8_t *src_ptr, int32_t src_stride, CALC_MSE_B(src, ref, var); } + for (; res--; ) { + v16u8 reg0; + v8i16 tmp0; + src0 = LW(src_ptr); + ref0 = LW(ref_ptr); + src_ptr += src_stride; + ref_ptr += ref_stride; + src = (v16u8)__msa_insert_w((v4i32) src, 0, src0); + ref = (v16u8)__msa_insert_w((v4i32) ref, 0, ref0); + reg0 = (v16u8)__msa_ilvr_b(src, ref); + reg0 = (v16u8)__msa_ilvr_d((v2i64) zero, (v2i64) reg0); + tmp0 = (v8i16)__msa_hsub_u_h((v16u8) reg0, (v16u8) reg0); + var = (v4i32)__msa_dpadd_s_w((v4i32) var, (v8i16) tmp0, (v8i16) tmp0); + } sse = HADD_SW_S32(var); return sse; @@ -435,13 +560,14 @@ static uint32_t sse_8width_msa(const uint8_t *src_ptr, int32_t src_stride, const uint8_t *ref_ptr, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 2; + int32_t res = height & 0x03; uint32_t sse; v16u8 src0, src1, src2, src3; v16u8 ref0, ref1, ref2, ref3; v4i32 var = { 0 }; - for (ht_cnt = (height >> 2); ht_cnt--;) { + for (; ht_cnt--; ) { LD_UB4(src_ptr, src_stride, src0, src1, src2, src3); src_ptr += (4 * src_stride); LD_UB4(ref_ptr, ref_stride, ref0, ref1, ref2, ref3); @@ -453,6 +579,16 @@ static uint32_t sse_8width_msa(const uint8_t *src_ptr, int32_t src_stride, CALC_MSE_B(src1, ref1, var); } + for (; res--; ) { + v8i16 tmp0; + src0 = LD_UB(src_ptr); + ref0 = LD_UB(ref_ptr); + src_ptr += src_stride; + ref_ptr += ref_stride; + ref1 = (v16u8)__msa_ilvr_b(src0, ref0); + tmp0 = (v8i16)__msa_hsub_u_h((v16u8) ref1, (v16u8) ref1); + var = (v4i32)__msa_dpadd_s_w((v4i32) var, (v8i16) tmp0, (v8i16) tmp0); + } sse = HADD_SW_S32(var); return sse; @@ -462,12 +598,13 @@ static uint32_t sse_16width_msa(const uint8_t *src_ptr, int32_t src_stride, const uint8_t *ref_ptr, int32_t ref_stride, int32_t height) { - int32_t ht_cnt; + int32_t ht_cnt = height >> 2; + int32_t res = height & 0x03; uint32_t sse; v16u8 src, ref; v4i32 var = { 0 }; - for (ht_cnt = (height >> 2); ht_cnt--;) { + for (; ht_cnt--; ) { src = LD_UB(src_ptr); src_ptr += src_stride; ref = LD_UB(ref_ptr); @@ -493,6 +630,14 @@ static uint32_t sse_16width_msa(const uint8_t *src_ptr, int32_t src_stride, CALC_MSE_B(src, ref, var); } + for (; res--; ) { + src = LD_UB(src_ptr); + src_ptr += src_stride; + ref = LD_UB(ref_ptr); + ref_ptr += ref_stride; + CALC_MSE_B(src, ref, var); + } + sse = HADD_SW_S32(var); return sse; @@ -544,7 +689,7 @@ static int32_t hadamard_diff_8x8_msa(const uint8_t *src, int32_t src_stride, } static int32_t hadamard_intra_8x8_msa(const uint8_t *src, int32_t src_stride, - const uint8_t *ref, int32_t ref_stride) + const uint8_t *dumy, int32_t ref_stride) { int32_t sum_res = 0; v16u8 src0, src1, src2, src3, src4, src5, src6, src7; @@ -659,10 +804,10 @@ int ff_hadamard8_diff8x8_msa(MpegEncContext *s, const uint8_t *dst, const uint8_ return hadamard_diff_8x8_msa(src, stride, dst, stride); } -int ff_hadamard8_intra8x8_msa(MpegEncContext *s, const uint8_t *dst, const uint8_t *src, +int ff_hadamard8_intra8x8_msa(MpegEncContext *s, const uint8_t *src, const uint8_t *dummy, ptrdiff_t stride, int h) { - return hadamard_intra_8x8_msa(src, stride, dst, stride); + return hadamard_intra_8x8_msa(src, stride, dummy, stride); } /* Hadamard Transform functions */ From patchwork Thu Sep 8 12:55:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6ZmI5piK?= X-Patchwork-Id: 37763 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:139a:b0:8f:1db5:eae2 with SMTP id w26csp268346pzh; Thu, 8 Sep 2022 05:56:41 -0700 (PDT) X-Google-Smtp-Source: AA6agR6fv8ylArQUYySA+7SZkfIJ6Nv1Kb6dVqQUCSBjthJ7TZlzb+7W0Tkjgic5kbAz+IYDjQ8G X-Received: by 2002:a17:907:a077:b0:770:8288:39e8 with SMTP id ia23-20020a170907a07700b00770828839e8mr5617699ejc.204.1662641801137; Thu, 08 Sep 2022 05:56:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1662641801; cv=none; d=google.com; s=arc-20160816; b=wyBNiCNueeaUVM3ZKPlI0WEn3kl+FZEimEuQeGMV/cGhWkNaHz53igCyq43izeNigs HG+MdinLeUaHuYpvXeBPz+tbmouAIKuNQzFvp5YYa3YSX0uofxK0Obp5POlg3PqCK+H4 cV1y0b2gJPhj5fQR4fYL82XtHqTatfH/wr+Ic5gfZgeDJi6SsqZS/Cavep9GcomaljYn yCshVvXsqhvoMSqW3iEulhehGYIgPbySkXXux1vHBTvmTg6PzeY2hbOhO0p6DMzPOcTR v4eGKfmbI8n3LuVf8Wj5d7V3Xmakopd+/bjPviPJwF+kTDn4wy8fHcENnBRvtc+nB7Ir IcbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=0jZI3HECOAJqOsUUkbNRCSLMULztV/KbMMoUTW2Hp5A=; b=puLRRXlXvB5jVWa2wnKfuqXQ7X7wPjwxI9h3tB2G/4ZltoCiVFcRy5PxzevlGS6tlu ew3JED20SczC5rfdosiCZ/xM39szK+t5rQnE9UHcieyDLpiy9aXrmqmGQGk2E8n7XQ/J nX+2lvTAp5OzjGzFw/695uwFKs9oMTO6gdVTtyzwQS28QA1EsGZFZljrx/D0CSLJJoA6 ZsiuAQqn6TXdez1trGiAWML2KfHxsPhJrOsINRJa+rX8nMEfdY+h791ENYXrUB3cNGRt wnmjGW8Bes/PkK2TuRdkAT+IguDwpkn7kxJcu+KMIRfQZ7tMuQ4YZHA730bXelEkqbrk d2AQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r11-20020a056402034b00b004477899842dsi8772630edw.499.2022.09.08.05.56.40; Thu, 08 Sep 2022 05:56:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6E93B68BAB4; Thu, 8 Sep 2022 15:56:19 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from loongson.cn (mail.loongson.cn [114.242.206.163]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C987E68BAAD for ; Thu, 8 Sep 2022 15:56:11 +0300 (EEST) Received: from localhost (unknown [36.33.26.144]) by localhost.localdomain (Coremail) with SMTP id AQAAf8AxFeI65hljnXEUAA--.18424S3; Thu, 08 Sep 2022 20:55:22 +0800 (CST) From: Hao Chen To: ffmpeg-devel@ffmpeg.org Date: Thu, 8 Sep 2022 20:55:19 +0800 Message-Id: <20220908125519.15807-3-chenhao@loongson.cn> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220908125519.15807-1-chenhao@loongson.cn> References: <20220908125519.15807-1-chenhao@loongson.cn> MIME-Version: 1.0 X-CM-TRANSID: AQAAf8AxFeI65hljnXEUAA--.18424S3 X-Coremail-Antispam: 1UD129KBjvAXoWDuw45uFW8CrW7XFWxtFWxtFb_yoWfJF18Jo Wvyas7trZ7CrySkrnxXr1Yya48Kr4UtF18ta4I9r1YqasaqFnYyas8Zw17Ar4ru3yxu3W5 CF9xKryvvw1ayrs7n29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7v73VFW2AGmfu7bjvjm3 AaLaJ3UjIYCTnIWjp_UUU5V7k0a2IF6w4kM7kC6x804xWl14x267AKxVWUJVW8JwAFc2x0 x2IEx4CE42xK8VAvwI8IcIk0rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2ocxC64kIII0Yj4 1l84x0c7CEw4AK67xGY2AK021l84ACjcxK6xIIjxv20xvE14v26ryj6F1UM28EF7xvwVC0 I7IYx2IY6xkF7I0E14v26F4j6r4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwV C2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC 0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUAVWUtwAv7VC2z280aVAFwI0_Gr0_Cr 1lOx8S6xCaFVCjc4AY6r1j6r4UMxkIecxEwVAFwVW5JwCFx2IqxVCFs4IE7xkEbVWUJVW8 JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1V AFwI0_Jr0_JrylIxAIcVC0I7IYx2IY67AKxVWUCVW8JwCI42IY6xIIjxv20xvEc7CjxVAF wI0_Jr0_Gr1lIxAIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r 4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5izV5UU UUU== X-CM-SenderInfo: hfkh0xtdr6z05rqj20fqof0/ Subject: [FFmpeg-devel] [PATCH v1 2/2] Fix hevc decoding bugs on MIPS paltform. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: q66RMOQn61cA The patch fixes the bugs that occurred when running fate-checkasm-hevc_pel on MIPS paltform. --- libavcodec/mips/hevc_macros_msa.h | 9 + libavcodec/mips/hevc_mc_bi_msa.c | 149 + libavcodec/mips/hevc_mc_biw_msa.c | 140 +- libavcodec/mips/hevc_mc_uni_msa.c | 89 + libavcodec/mips/hevc_mc_uniw_msa.c | 4176 +++++++++++++++++----------- libavcodec/mips/hevcdsp_msa.c | 86 +- 6 files changed, 3021 insertions(+), 1628 deletions(-) diff --git a/libavcodec/mips/hevc_macros_msa.h b/libavcodec/mips/hevc_macros_msa.h index ea53812b64..02c96b752b 100644 --- a/libavcodec/mips/hevc_macros_msa.h +++ b/libavcodec/mips/hevc_macros_msa.h @@ -52,6 +52,15 @@ out_m; \ } ) +#define HEVC_FILT_4TAP_SW(in0, in1, filt0, filt1) \ +( { \ + v4i32 out_m; \ + \ + out_m = __msa_dotp_s_w((v8i16) in0, (v8i16) filt0); \ + out_m = __msa_dpadd_s_w(out_m, (v8i16) in1, (v8i16) filt1); \ + out_m; \ +} ) + #define HEVC_FILT_4TAP(in0, in1, filt0, filt1) \ ( { \ v4i32 out_m; \ diff --git a/libavcodec/mips/hevc_mc_bi_msa.c b/libavcodec/mips/hevc_mc_bi_msa.c index 701e12ab86..dac6a32ab4 100644 --- a/libavcodec/mips/hevc_mc_bi_msa.c +++ b/libavcodec/mips/hevc_mc_bi_msa.c @@ -136,6 +136,7 @@ static void hevc_bi_copy_6w_msa(const uint8_t *src0_ptr, { uint32_t loop_cnt; uint64_t tp0, tp1, tp2, tp3; + int32_t res = height & 0x07; v16u8 out0, out1, out2, out3; v16i8 zero = { 0 }; v16i8 src0 = { 0 }, src1 = { 0 }, src2 = { 0 }, src3 = { 0 }; @@ -176,6 +177,45 @@ static void hevc_bi_copy_6w_msa(const uint8_t *src0_ptr, ST_H2(out3, 2, 6, dst + 2 * dst_stride + 4, dst_stride); dst += (4 * dst_stride); } + if (res) { + LD4(src0_ptr, src_stride, tp0, tp1, tp2, tp3); + src0_ptr += (4 * src_stride); + INSERT_D2_SB(tp0, tp1, src0); + INSERT_D2_SB(tp2, tp3, src1); + LD4(src0_ptr, src_stride, tp0, tp1, tp2, tp3); + INSERT_D2_SB(tp0, tp1, src2); + INSERT_D2_SB(tp2, tp3, src3); + LD_SH8(src1_ptr, src2_stride, in0, in1, in2, in3, in4, in5, in6, in7); + ILVRL_B2_SH(zero, src0, dst0, dst1); + ILVRL_B2_SH(zero, src1, dst2, dst3); + ILVRL_B2_SH(zero, src2, dst4, dst5); + ILVRL_B2_SH(zero, src3, dst6, dst7); + SLLI_4V(dst0, dst1, dst2, dst3, 6); + SLLI_4V(dst4, dst5, dst6, dst7, 6); + HEVC_BI_RND_CLIP4_MAX_SATU(in0, in1, in2, in3, dst0, dst1, dst2, dst3, + 7, dst0, dst1, dst2, dst3); + HEVC_BI_RND_CLIP4_MAX_SATU(in4, in5, in6, in7, dst4, dst5, dst6, dst7, + 7, dst4, dst5, dst6, dst7); + PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + if (res == 2) { + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + } else if (res == 4) { + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); + ST_H2(out1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); + } else { + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); + ST_H2(out1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); + dst += (4 * dst_stride); + ST_W2(out2, 0, 2, dst, dst_stride); + ST_H2(out2, 2, 6, dst + 4, dst_stride); + } + } } static void hevc_bi_copy_8w_msa(const uint8_t *src0_ptr, @@ -536,6 +576,7 @@ static void hevc_hz_bi_8t_4w_msa(const uint8_t *src0_ptr, int32_t height) { uint32_t loop_cnt; + int32_t res = height & 0x07; v8i16 filt0, filt1, filt2, filt3; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v16i8 mask1, mask2, mask3; @@ -597,6 +638,50 @@ static void hevc_hz_bi_8t_4w_msa(const uint8_t *src0_ptr, ST_W8(dst0, dst1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); } + if (res) { + LD_SB8(src0_ptr, src_stride, src0, src1, src2, src3, + src4, src5, src6, src7); + LD_SH8(src1_ptr, src2_stride, in0, in1, in2, in3, in4, in5, in6, in7); + + ILVR_D2_SH(in1, in0, in3, in2, in0, in1); + ILVR_D2_SH(in5, in4, in7, in6, in2, in3); + XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); + + dst0 = const_vec; + dst1 = const_vec; + dst2 = const_vec; + dst3 = const_vec; + VSHF_B2_SB(src0, src1, src2, src3, mask0, mask0, vec0, vec1); + VSHF_B2_SB(src4, src5, src6, src7, mask0, mask0, vec2, vec3); + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt0, filt0, filt0, dst0, + dst1, dst2, dst3); + VSHF_B2_SB(src0, src1, src2, src3, mask1, mask1, vec0, vec1); + VSHF_B2_SB(src4, src5, src6, src7, mask1, mask1, vec2, vec3); + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt1, filt1, filt1, filt1, dst0, + dst1, dst2, dst3); + VSHF_B2_SB(src0, src1, src2, src3, mask2, mask2, vec0, vec1); + VSHF_B2_SB(src4, src5, src6, src7, mask2, mask2, vec2, vec3); + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt2, filt2, filt2, filt2, dst0, + dst1, dst2, dst3); + VSHF_B2_SB(src0, src1, src2, src3, mask3, mask3, vec0, vec1); + VSHF_B2_SB(src4, src5, src6, src7, mask3, mask3, vec2, vec3); + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt3, filt3, filt3, filt3, dst0, + dst1, dst2, dst3); + + HEVC_BI_RND_CLIP4(in0, in1, in2, in3, + dst0, dst1, dst2, dst3, 7, dst0, dst1, dst2, dst3); + + PCKEV_B2_SH(dst1, dst0, dst3, dst2, dst0, dst1); + if (res == 2) { + ST_W2(dst0, 0, 1, dst, dst_stride); + } else if (res == 4) { + ST_W4(dst0, 0, 1, 2, 3, dst, dst_stride); + } else { + ST_W4(dst0, 0, 1, 2, 3, dst, dst_stride); + dst += (4 * dst_stride); + ST_W2(dst1, 0, 1, dst, dst_stride); + } + } } static void hevc_hz_bi_8t_8w_msa(const uint8_t *src0_ptr, @@ -1182,6 +1267,7 @@ static void hevc_vt_bi_8t_4w_msa(const uint8_t *src0_ptr, int32_t height) { int32_t loop_cnt; + int32_t res = height & 0x07; v16i8 src0, src1, src2, src3, src4, src5; v16i8 src6, src7, src8, src9, src10; v8i16 in0, in1, in2, in3, in4, in5, in6, in7; @@ -1256,6 +1342,50 @@ static void hevc_vt_bi_8t_4w_msa(const uint8_t *src0_ptr, src6554 = src14131312; src6 = src14; } + if (res) { + LD_SB8(src0_ptr, src_stride, + src7, src8, src9, src10, src11, src12, src13, src14); + LD_SH8(src1_ptr, src2_stride, in0, in1, in2, in3, in4, in5, in6, in7); + + ILVR_D2_SH(in1, in0, in3, in2, in0, in1); + ILVR_D2_SH(in5, in4, in7, in6, in2, in3); + ILVR_B4_SB(src7, src6, src8, src7, src9, src8, src10, src9, + src76_r, src87_r, src98_r, src109_r); + ILVR_B4_SB(src11, src10, src12, src11, src13, src12, src14, src13, + src1110_r, src1211_r, src1312_r, src1413_r); + ILVR_D4_SB(src87_r, src76_r, src109_r, src98_r, src1211_r, src1110_r, + src1413_r, src1312_r, + src8776, src10998, src12111110, src14131312); + XORI_B4_128_SB(src8776, src10998, src12111110, src14131312); + + dst10 = const_vec; + DPADD_SB4_SH(src2110, src4332, src6554, src8776, + filt0, filt1, filt2, filt3, dst10, dst10, dst10, dst10); + dst32 = const_vec; + DPADD_SB4_SH(src4332, src6554, src8776, src10998, + filt0, filt1, filt2, filt3, dst32, dst32, dst32, dst32); + dst54 = const_vec; + DPADD_SB4_SH(src6554, src8776, src10998, src12111110, + filt0, filt1, filt2, filt3, dst54, dst54, dst54, dst54); + dst76 = const_vec; + DPADD_SB4_SH(src8776, src10998, src12111110, src14131312, + filt0, filt1, filt2, filt3, dst76, dst76, dst76, dst76); + + HEVC_BI_RND_CLIP4(in0, in1, in2, in3, + dst10, dst32, dst54, dst76, 7, + dst10, dst32, dst54, dst76); + + PCKEV_B2_SH(dst32, dst10, dst76, dst54, dst10, dst54); + if (res == 2) { + ST_W2(dst10, 0, 1, dst, dst_stride); + } else if (res == 4) { + ST_W4(dst10, 0, 1, 2, 3, dst, dst_stride); + } else { + ST_W4(dst10, 0, 1, 2, 3, dst, dst_stride); + dst += 4 * dst_stride; + ST_W2(dst54, 0, 1, dst, dst_stride); + } + } } static void hevc_vt_bi_8t_8w_msa(const uint8_t *src0_ptr, @@ -2363,6 +2493,7 @@ static void hevc_hz_bi_4t_6w_msa(const uint8_t *src0_ptr, int32_t height) { uint32_t loop_cnt; + int32_t res = height & 0x03; v8i16 filt0, filt1; v16i8 src0, src1, src2, src3; v8i16 in0, in1, in2, in3; @@ -2412,6 +2543,24 @@ static void hevc_hz_bi_4t_6w_msa(const uint8_t *src0_ptr, ST_H2(dst1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); dst += (4 * dst_stride); } + if (res) { + LD_SB2(src0_ptr, src_stride, src0, src1); + LD_SH2(src1_ptr, src2_stride, in0, in1); + XORI_B2_128_SB(src0, src1); + + dst0 = const_vec; + dst1 = const_vec; + VSHF_B2_SB(src0, src0, src1, src1, mask0, mask0, vec0, vec1); + DPADD_SB2_SH(vec0, vec1, filt0, filt0, dst0, dst1); + VSHF_B2_SB(src0, src0, src1, src1, mask1, mask1, vec0, vec1); + DPADD_SB2_SH(vec0, vec1, filt1, filt1, dst0, dst1); + + HEVC_BI_RND_CLIP2(in0, in1, dst0, dst1, 7, dst0, dst1); + + dst0 = (v8i16) __msa_pckev_b((v16i8) dst1, (v16i8) dst0); + ST_W2(dst0, 0, 2, dst, dst_stride); + ST_H2(dst0, 2, 6, dst + 4, dst_stride); + } } static void hevc_hz_bi_4t_8x2_msa(const uint8_t *src0_ptr, diff --git a/libavcodec/mips/hevc_mc_biw_msa.c b/libavcodec/mips/hevc_mc_biw_msa.c index 981c37d7e8..260ec84496 100644 --- a/libavcodec/mips/hevc_mc_biw_msa.c +++ b/libavcodec/mips/hevc_mc_biw_msa.c @@ -42,8 +42,8 @@ static const uint8_t ff_hevc_mask_arr[16 * 2] __attribute__((aligned(0x40))) = { out1_l = __msa_dpadd_s_w(offset, (v8i16) out1_l, (v8i16) wgt); \ \ SRAR_W4_SW(out0_r, out1_r, out0_l, out1_l, rnd); \ + CLIP_SW4_0_255(out0_l, out0_r, out1_l, out1_r); \ PCKEV_H2_SH(out0_l, out0_r, out1_l, out1_r, out0, out1); \ - CLIP_SH2_0_255(out0, out1); \ } #define HEVC_BIW_RND_CLIP4(in0, in1, in2, in3, vec0, vec1, vec2, vec3, \ @@ -65,8 +65,8 @@ static const uint8_t ff_hevc_mask_arr[16 * 2] __attribute__((aligned(0x40))) = { out0_l = __msa_dpadd_s_w(offset, (v8i16) out0_l, (v8i16) wgt); \ out1_l = __msa_dpadd_s_w(offset, (v8i16) out1_l, (v8i16) wgt); \ SRAR_W4_SW(out0_r, out1_r, out0_l, out1_l, rnd); \ + CLIP_SW4_0_255(out0_r, out1_r, out0_l, out1_l); \ PCKEV_H2_SH(out0_l, out0_r, out1_l, out1_r, out0, out1); \ - CLIP_SH2_0_255(out0, out1); \ } #define HEVC_BIW_RND_CLIP4_MAX_SATU(in0, in1, in2, in3, vec0, vec1, vec2, \ @@ -123,8 +123,8 @@ static void hevc_biwgt_copy_4w_msa(const uint8_t *src0_ptr, dst0_r = __msa_dpadd_s_w(offset_vec, (v8i16) dst0_r, weight_vec); dst0_l = __msa_dpadd_s_w(offset_vec, (v8i16) dst0_l, weight_vec); SRAR_W2_SW(dst0_r, dst0_l, rnd_vec); + CLIP_SW2_0_255(dst0_r, dst0_l); dst0 = (v8i16) __msa_pckev_h((v8i16) dst0_l, (v8i16) dst0_r); - CLIP_SH_0_255(dst0); out0 = (v16u8) __msa_pckev_b((v16i8) dst0, (v16i8) dst0); ST_W2(out0, 0, 1, dst, dst_stride); } else if (4 == height) { @@ -182,6 +182,7 @@ static void hevc_biwgt_copy_6w_msa(const uint8_t *src0_ptr, int32_t rnd_val) { uint32_t loop_cnt; + int32_t res = height & 0x03; int32_t offset, weight; uint64_t tp0, tp1, tp2, tp3; v16u8 out0, out1; @@ -220,6 +221,27 @@ static void hevc_biwgt_copy_6w_msa(const uint8_t *src0_ptr, ST_H2(out1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); dst += (4 * dst_stride); } + if (res) { + LD4(src0_ptr, src_stride, tp0, tp1, tp2, tp3); + src0_ptr += (4 * src_stride); + INSERT_D2_SB(tp0, tp1, src0); + INSERT_D2_SB(tp2, tp3, src1); + LD_SH4(src1_ptr, src2_stride, in0, in1, in2, in3); + src1_ptr += (4 * src2_stride); + ILVRL_B2_SH(zero, src0, dst0, dst1); + ILVRL_B2_SH(zero, src1, dst2, dst3); + SLLI_4V(dst0, dst1, dst2, dst3, 6); + HEVC_BIW_RND_CLIP4_MAX_SATU(dst0, dst1, dst2, dst3, + in0, in1, in2, in3, + weight_vec, rnd_vec, offset_vec, + dst0, dst1, dst2, dst3); + + PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); + ST_H2(out1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); + } } static void hevc_biwgt_copy_8w_msa(const uint8_t *src0_ptr, @@ -340,7 +362,7 @@ static void hevc_biwgt_copy_12w_msa(const uint8_t *src0_ptr, weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val + 1); - for (loop_cnt = (16 >> 2); loop_cnt--;) { + for (loop_cnt = (height >> 2); loop_cnt--;) { LD_SB4(src0_ptr, src_stride, src0, src1, src2, src3); src0_ptr += (4 * src_stride); LD_SH4(src1_ptr, src2_stride, in0, in1, in2, in3); @@ -1069,8 +1091,8 @@ static void hevc_hz_biwgt_8t_24w_msa(const uint8_t *src0_ptr, dst2_l = __msa_dpadd_s_w(offset_vec, (v8i16) dst2_l, (v8i16) weight_vec); SRAR_W2_SW(dst2_r, dst2_l, rnd_vec); + CLIP_SW2_0_255(dst2_r, dst2_l); out2 = __msa_pckev_h((v8i16) dst2_l, (v8i16) dst2_r); - CLIP_SH_0_255(out2); LD_SB2(src0_ptr, 16, src0, src1); src0_ptr += src_stride; @@ -1100,8 +1122,8 @@ static void hevc_hz_biwgt_8t_24w_msa(const uint8_t *src0_ptr, dst2_r = __msa_dpadd_s_w(offset_vec, (v8i16) dst2_r, (v8i16) weight_vec); dst2_l = __msa_dpadd_s_w(offset_vec, (v8i16) dst2_l, (v8i16) weight_vec); SRAR_W2_SW(dst2_r, dst2_l, rnd_vec); + CLIP_SW2_0_255(dst2_r, dst2_l); out2 = __msa_pckev_h((v8i16) dst2_l, (v8i16) dst2_r); - CLIP_SH_0_255(out2); PCKEV_B2_SH(out1, out0, out2, out2, out0, out2); dst_val0 = __msa_copy_u_d((v2i64) out2, 0); ST_SH(out0, dst); @@ -1413,6 +1435,7 @@ static void hevc_vt_biwgt_8t_4w_msa(const uint8_t *src0_ptr, int32_t rnd_val) { uint32_t loop_cnt; + int32_t res = height & 0x07; int32_t offset, weight; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 src11, src12, src13, src14; @@ -1489,6 +1512,46 @@ static void hevc_vt_biwgt_8t_4w_msa(const uint8_t *src0_ptr, ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); + src2110 = src10998; + src4332 = src12111110; + src6554 = src14131312; + src6 = src14; + } + if (res) { + LD_SB8(src0_ptr, src_stride, + src7, src8, src9, src10, src11, src12, src13, src14); + src0_ptr += (8 * src_stride); + LD_SH8(src1_ptr, src2_stride, in0, in1, in2, in3, in4, in5, in6, in7); + src1_ptr += (8 * src2_stride); + + ILVR_D2_SH(in1, in0, in3, in2, in0, in1); + ILVR_D2_SH(in5, in4, in7, in6, in2, in3); + ILVR_B4_SB(src7, src6, src8, src7, src9, src8, src10, src9, + src76_r, src87_r, src98_r, src109_r); + ILVR_B4_SB(src11, src10, src12, src11, src13, src12, src14, src13, + src1110_r, src1211_r, src1312_r, src1413_r); + ILVR_D4_SB(src87_r, src76_r, src109_r, src98_r, src1211_r, src1110_r, + src1413_r, src1312_r, + src8776, src10998, src12111110, src14131312); + XORI_B4_128_SB(src8776, src10998, src12111110, src14131312); + + DOTP_SB4_SH(src2110, src4332, src6554, src8776, filt0, filt0, filt0, + filt0, dst10, dst32, dst54, dst76); + DPADD_SB4_SH(src4332, src6554, src8776, src10998, filt1, filt1, filt1, + filt1, dst10, dst32, dst54, dst76); + DPADD_SB4_SH(src6554, src8776, src10998, src12111110, filt2, filt2, + filt2, filt2, dst10, dst32, dst54, dst76); + DPADD_SB4_SH(src8776, src10998, src12111110, src14131312, filt3, filt3, + filt3, filt3, dst10, dst32, dst54, dst76); + + HEVC_BIW_RND_CLIP4(dst10, dst32, dst54, dst76, + in0, in1, in2, in3, + weight_vec, rnd_vec, offset_vec, + out0, out1, out2, out3); + + PCKEV_B2_SH(out1, out0, out3, out2, out0, out1); + ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); + src2110 = src10998; src4332 = src12111110; src6554 = src14131312; @@ -1674,8 +1737,8 @@ static void hevc_vt_biwgt_8t_12w_msa(const uint8_t *src0_ptr, dst2_l = __msa_dpadd_s_w(offset_vec, (v8i16) dst2_l, (v8i16) weight_vec); SRAR_W2_SW(dst2_r, dst2_l, rnd_vec); + CLIP_SW2_0_255(dst2_r, dst2_l); out2 = __msa_pckev_h((v8i16) dst2_l, (v8i16) dst2_r); - CLIP_SH_0_255(out2); PCKEV_B2_SH(out1, out0, out2, out2, out0, out2); ST_D2(out0, 0, 1, dst, dst_stride); ST_W2(out2, 0, 1, dst + 8, dst_stride); @@ -3555,8 +3618,8 @@ static void hevc_vt_biwgt_4t_4x2_msa(const uint8_t *src0_ptr, dst10_r = __msa_dpadd_s_w(offset_vec, (v8i16) dst10_r, (v8i16) weight_vec); dst10_l = __msa_dpadd_s_w(offset_vec, (v8i16) dst10_l, (v8i16) weight_vec); SRAR_W2_SW(dst10_r, dst10_l, rnd_vec); + CLIP_SW2_0_255(dst10_r, dst10_l); out = __msa_pckev_h((v8i16) dst10_l, (v8i16) dst10_r); - CLIP_SH_0_255(out); out = (v8i16) __msa_pckev_b((v16i8) out, (v16i8) out); ST_W2(out, 0, 1, dst, dst_stride); } @@ -3760,6 +3823,7 @@ static void hevc_vt_biwgt_4t_6w_msa(const uint8_t *src0_ptr, int32_t rnd_val) { uint32_t loop_cnt; + int32_t res = height & 0x03; int32_t offset, weight, constant; v16i8 src0, src1, src2, src3, src4; v8i16 in0, in1, in2, in3; @@ -3820,6 +3884,35 @@ static void hevc_vt_biwgt_4t_6w_msa(const uint8_t *src0_ptr, ST_H2(tmp1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); dst += (4 * dst_stride); } + if (res) { + LD_SB2(src0_ptr, src_stride, src3, src4); + src0_ptr += (2 * src_stride); + LD_SH4(src1_ptr, src2_stride, in0, in1, in2, in3); + src1_ptr += (4 * src2_stride); + XORI_B2_128_SB(src3, src4); + ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); + + tmp0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); + tmp1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); + + LD_SB2(src0_ptr, src_stride, src1, src2); + src0_ptr += (2 * src_stride); + XORI_B2_128_SB(src1, src2); + ILVR_B2_SB(src1, src4, src2, src1, src10_r, src21_r); + + tmp2 = HEVC_FILT_4TAP_SH(src32_r, src10_r, filt0, filt1); + tmp3 = HEVC_FILT_4TAP_SH(src43_r, src21_r, filt0, filt1); + HEVC_BIW_RND_CLIP4(tmp0, tmp1, tmp2, tmp3, + in0, in1, in2, in3, + weight_vec, rnd_vec, offset_vec, + tmp0, tmp1, tmp2, tmp3); + + PCKEV_B2_SH(tmp1, tmp0, tmp3, tmp2, tmp0, tmp1); + ST_W2(tmp0, 0, 2, dst, dst_stride); + ST_H2(tmp0, 2, 6, dst + 4, dst_stride); + ST_W2(tmp1, 0, 2, dst + 2 * dst_stride, dst_stride); + ST_H2(tmp1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); + } } static void hevc_vt_biwgt_4t_8x2_msa(const uint8_t *src0_ptr, @@ -4575,8 +4668,8 @@ static void hevc_hv_biwgt_4t_4x2_msa(const uint8_t *src0_ptr, dst0 = __msa_dpadd_s_w(offset_vec, tmp0, weight_vec); dst1 = __msa_dpadd_s_w(offset_vec, tmp1, weight_vec); SRAR_W2_SW(dst0, dst1, rnd_vec); + CLIP_SW2_0_255(dst0, dst1); tmp = __msa_pckev_h((v8i16) dst1, (v8i16) dst0); - CLIP_SH_0_255(tmp); out = (v16u8) __msa_pckev_b((v16i8) tmp, (v16i8) tmp); ST_W2(out, 0, 1, dst, dst_stride); } @@ -4672,8 +4765,8 @@ static void hevc_hv_biwgt_4t_4x4_msa(const uint8_t *src0_ptr, dst2 = __msa_dpadd_s_w(offset_vec, tmp2, weight_vec); dst3 = __msa_dpadd_s_w(offset_vec, tmp3, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); - CLIP_SH2_0_255(tmp0, tmp1); out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_W4(out, 0, 1, 2, 3, dst, dst_stride); } @@ -4809,9 +4902,10 @@ static void hevc_hv_biwgt_4t_4multx8mult_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); @@ -5007,9 +5101,10 @@ static void hevc_hv_biwgt_4t_6w_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); @@ -5029,9 +5124,9 @@ static void hevc_hv_biwgt_4t_6w_msa(const uint8_t *src0_ptr, dst2 = __msa_dpadd_s_w(offset_vec, tmp2, weight_vec); dst3 = __msa_dpadd_s_w(offset_vec, tmp3, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp4, tmp5); - CLIP_SH2_0_255(tmp4, tmp5); out2 = (v16u8) __msa_pckev_b((v16i8) tmp5, (v16i8) tmp4); ST_H8(out2, 0, 1, 2, 3, 4, 5, 6, 7, dst + 4, dst_stride); } @@ -5126,8 +5221,8 @@ static void hevc_hv_biwgt_4t_8x2_msa(const uint8_t *src0_ptr, dst1_r = __msa_dpadd_s_w(offset_vec, tmp2, weight_vec); dst1_l = __msa_dpadd_s_w(offset_vec, tmp3, weight_vec); SRAR_W4_SW(dst0_r, dst0_l, dst1_r, dst1_l, rnd_vec); + CLIP_SW4_0_255(dst0_r, dst0_l, dst1_r, dst1_l); PCKEV_H2_SH(dst0_l, dst0_r, dst1_l, dst1_r, tmp0, tmp1); - CLIP_SH2_0_255(tmp0, tmp1); out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_D2(out, 0, 1, dst, dst_stride); } @@ -5247,9 +5342,10 @@ static void hevc_hv_biwgt_4t_8multx4_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); dst += 8; @@ -5386,9 +5482,10 @@ static void hevc_hv_biwgt_4t_8x6_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); PCKEV_H2_SW(dst4_l, dst4_r, dst5_l, dst5_r, dst0, dst1); @@ -5399,8 +5496,8 @@ static void hevc_hv_biwgt_4t_8x6_msa(const uint8_t *src0_ptr, dst2 = __msa_dpadd_s_w(offset_vec, tmp2, weight_vec); dst3 = __msa_dpadd_s_w(offset_vec, tmp3, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp4, tmp5); - CLIP_SH2_0_255(tmp4, tmp5); out2 = (v16u8) __msa_pckev_b((v16i8) tmp5, (v16i8) tmp4); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_D2(out2, 0, 1, dst + 4 * dst_stride, dst_stride); @@ -5536,9 +5633,10 @@ static void hevc_hv_biwgt_4t_8multx4mult_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst_tmp, dst_stride); dst_tmp += (4 * dst_stride); @@ -5724,9 +5822,10 @@ static void hevc_hv_biwgt_4t_12w_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst_tmp, dst_stride); dst_tmp += (4 * dst_stride); @@ -5820,9 +5919,10 @@ static void hevc_hv_biwgt_4t_12w_msa(const uint8_t *src0_ptr, dst7 = __msa_dpadd_s_w(offset_vec, tmp7, weight_vec); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); diff --git a/libavcodec/mips/hevc_mc_uni_msa.c b/libavcodec/mips/hevc_mc_uni_msa.c index c0571e1231..e221df7d53 100644 --- a/libavcodec/mips/hevc_mc_uni_msa.c +++ b/libavcodec/mips/hevc_mc_uni_msa.c @@ -908,6 +908,7 @@ static void common_vt_8t_4w_msa(const uint8_t *src, int32_t src_stride, const int8_t *filter, int32_t height) { uint32_t loop_cnt; + uint32_t res = (height & 0x07) >> 1; v16u8 out0, out1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 src11, src12, src13, src14; @@ -970,6 +971,27 @@ static void common_vt_8t_4w_msa(const uint8_t *src, int32_t src_stride, src6554 = src14131312; src6 = src14; } + for (; res--; ) { + LD_SB2(src, src_stride, src7, src8); + src += 2 * src_stride; + ILVR_B2_SB(src7, src6, src8, src7, src76_r, src87_r); + src8776 = (v16i8)__msa_ilvr_d((v2i64) src87_r, (v2i64) src76_r); + src8776 = (v16i8)__msa_xori_b(src8776, 128); + out10 = (v8i16)__msa_dotp_s_h((v16i8) src2110, (v16i8) filt0); + out10 = (v8i16)__msa_dpadd_s_h((v8i16) out10, src4332, filt1); + out10 = (v8i16)__msa_dpadd_s_h((v8i16) out10, src6554, filt2); + out10 = (v8i16)__msa_dpadd_s_h((v8i16) out10, src8776, filt3); + out10 = (v8i16)__msa_srari_h((v8i16) out10, 6); + out10 = (v8i16)__msa_sat_s_h((v8i16) out10, 7); + out0 = (v16u8)__msa_pckev_b((v16i8) out10, (v16i8) out10); + out0 = (v16u8)__msa_xori_b((v16u8) out0, 128); + ST_W2(out0, 0, 1, dst, dst_stride); + dst += 2 * dst_stride; + src2110 = src4332; + src4332 = src6554; + src6554 = src8776; + src6 = src8; + } } static void common_vt_8t_8w_msa(const uint8_t *src, int32_t src_stride, @@ -1341,6 +1363,7 @@ static void hevc_hv_uni_8t_4w_msa(const uint8_t *src, int32_t height) { uint32_t loop_cnt; + uint32_t res = height & 0x07; v16u8 out0, out1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; v16i8 src9, src10, src11, src12, src13, src14; @@ -1464,6 +1487,72 @@ static void hevc_hv_uni_8t_4w_msa(const uint8_t *src, dst65_r = dst1413_r; dst66 = (v8i16) __msa_splati_d((v2i64) dst1410, 1); } + if (res) { + LD_SB8(src, src_stride, src7, src8, src9, src10, src11, src12, src13, + src14); + XORI_B8_128_SB(src7, src8, src9, src10, src11, src12, src13, src14); + + VSHF_B4_SB(src7, src11, mask0, mask1, mask2, mask3, + vec0, vec1, vec2, vec3); + VSHF_B4_SB(src8, src12, mask0, mask1, mask2, mask3, + vec4, vec5, vec6, vec7); + VSHF_B4_SB(src9, src13, mask0, mask1, mask2, mask3, + vec8, vec9, vec10, vec11); + VSHF_B4_SB(src10, src14, mask0, mask1, mask2, mask3, + vec12, vec13, vec14, vec15); + + dst117 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3); + dst128 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3); + dst139 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, + filt2, filt3); + dst1410 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, + filt2, filt3); + + dst76_r = __msa_ilvr_h(dst117, dst66); + ILVRL_H2_SH(dst128, dst117, dst87_r, dst1211_r); + ILVRL_H2_SH(dst139, dst128, dst98_r, dst1312_r); + ILVRL_H2_SH(dst1410, dst139, dst109_r, dst1413_r); + dst117 = (v8i16) __msa_splati_d((v2i64) dst117, 1); + dst1110_r = __msa_ilvr_h(dst117, dst1410); + + dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst1_r = HEVC_FILT_8TAP(dst21_r, dst43_r, dst65_r, dst87_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst2_r = HEVC_FILT_8TAP(dst32_r, dst54_r, dst76_r, dst98_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst3_r = HEVC_FILT_8TAP(dst43_r, dst65_r, dst87_r, dst109_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst4_r = HEVC_FILT_8TAP(dst54_r, dst76_r, dst98_r, dst1110_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst5_r = HEVC_FILT_8TAP(dst65_r, dst87_r, dst109_r, dst1211_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst6_r = HEVC_FILT_8TAP(dst76_r, dst98_r, dst1110_r, dst1312_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst7_r = HEVC_FILT_8TAP(dst87_r, dst109_r, dst1211_r, dst1413_r, + filt_h0, filt_h1, filt_h2, filt_h3); + + SRA_4V(dst0_r, dst1_r, dst2_r, dst3_r, 6); + SRA_4V(dst4_r, dst5_r, dst6_r, dst7_r, 6); + SRARI_W4_SW(dst0_r, dst1_r, dst2_r, dst3_r, 6); + SRARI_W4_SW(dst4_r, dst5_r, dst6_r, dst7_r, 6); + SAT_SW4_SW(dst0_r, dst1_r, dst2_r, dst3_r, 7); + SAT_SW4_SW(dst4_r, dst5_r, dst6_r, dst7_r, 7); + PCKEV_H2_SW(dst1_r, dst0_r, dst3_r, dst2_r, dst0_r, dst1_r); + PCKEV_H2_SW(dst5_r, dst4_r, dst7_r, dst6_r, dst4_r, dst5_r); + out0 = PCKEV_XORI128_UB(dst0_r, dst1_r); + out1 = PCKEV_XORI128_UB(dst4_r, dst5_r); + if (res == 2) { + ST_W2(out0, 0, 1, dst, dst_stride); + } else if(res == 4) { + ST_W4(out0, 0, 1, 2, 3, dst, dst_stride); + } else { + ST_W4(out0, 0, 1, 2, 3, dst, dst_stride); + ST_W2(out1, 0, 1, dst + 4 * dst_stride, dst_stride); + } + } } static void hevc_hv_uni_8t_8multx2mult_msa(const uint8_t *src, diff --git a/libavcodec/mips/hevc_mc_uniw_msa.c b/libavcodec/mips/hevc_mc_uniw_msa.c index 3c3d7a0da2..caf40c34da 100644 --- a/libavcodec/mips/hevc_mc_uniw_msa.c +++ b/libavcodec/mips/hevc_mc_uniw_msa.c @@ -33,15 +33,17 @@ static const uint8_t ff_hevc_mask_arr[16 * 2] __attribute__((aligned(0x40))) = { out0_h, out1_h) \ { \ v4i32 in0_r_m, in0_l_m, in1_r_m, in1_l_m; \ + v8i16 zero = { 0 }; \ \ - ILVRL_H2_SW(in0_h, in0_h, in0_r_m, in0_l_m); \ - ILVRL_H2_SW(in1_h, in1_h, in1_r_m, in1_l_m); \ - DOTP_SH4_SW(in0_r_m, in1_r_m, in0_l_m, in1_l_m, wgt_w, wgt_w, wgt_w, \ - wgt_w, in0_r_m, in1_r_m, in0_l_m, in1_l_m); \ - SRAR_W4_SW(in0_r_m, in1_r_m, in0_l_m, in1_l_m, rnd_w); \ + ILVRL_H2_SW(zero, in0_h, in0_r_m, in0_l_m); \ + ILVRL_H2_SW(zero, in1_h, in1_r_m, in1_l_m); \ + MUL4(in0_r_m, wgt_w, in0_l_m, wgt_w, in1_r_m, wgt_w, in1_l_m, wgt_w, \ + in0_r_m, in0_l_m, in1_r_m, in1_l_m); \ + SRAR_W4_SW(in0_r_m, in0_l_m, in1_r_m, in1_l_m, rnd_w); \ + ADD4(in0_r_m, offset_h, in0_l_m, offset_h, in1_r_m, offset_h, in1_l_m, \ + offset_h, in0_r_m, in0_l_m, in1_r_m, in1_l_m); \ + CLIP_SW4_0_255(in0_r_m, in0_l_m, in1_r_m, in1_l_m); \ PCKEV_H2_SH(in0_l_m, in0_r_m, in1_l_m, in1_r_m, out0_h, out1_h); \ - ADDS_SH2_SH(out0_h, offset_h, out1_h, offset_h, out0_h, out1_h); \ - CLIP_SH2_0_255(out0_h, out1_h); \ } #define HEVC_UNIW_RND_CLIP4_MAX_SATU_H(in0_h, in1_h, in2_h, in3_h, wgt_w, \ @@ -54,6 +56,21 @@ static const uint8_t ff_hevc_mask_arr[16 * 2] __attribute__((aligned(0x40))) = { out2_h, out3_h); \ } +#define HEVC_FILT_8TAP_4W_SH(in0, in1, in2, in3, filt0, filt1, \ + filt2, filt3, dst0, dst1) \ +{ \ + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; \ + ILVRL_B2_SH(zero, in0, tmp0, tmp4); \ + ILVRL_B2_SH(zero, in1, tmp1, tmp5); \ + ILVRL_B2_SH(zero, in2, tmp2, tmp6); \ + ILVRL_B2_SH(zero, in3, tmp3, tmp7); \ + dst0 = __msa_dotp_s_w((v8i16) tmp0, (v8i16) filt0); \ + dst1 = __msa_dotp_s_w((v8i16) tmp4, (v8i16) filt0); \ + DPADD_SH2_SW(tmp1, tmp5, filt1, filt1, dst0, dst1); \ + DPADD_SH2_SW(tmp2, tmp6, filt2, filt2, dst0, dst1); \ + DPADD_SH2_SW(tmp3, tmp7, filt3, filt3, dst0, dst1); \ +} + static void hevc_uniwgt_copy_4w_msa(const uint8_t *src, int32_t src_stride, uint8_t *dst, @@ -70,9 +87,8 @@ static void hevc_uniwgt_copy_4w_msa(const uint8_t *src, v8i16 dst0, dst1, dst2, dst3, offset_vec; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); if (2 == height) { @@ -83,12 +99,13 @@ static void hevc_uniwgt_copy_4w_msa(const uint8_t *src, dst0 = (v8i16) __msa_ilvr_b(zero, src0); dst0 <<= 6; - ILVRL_H2_SW(dst0, dst0, dst0_r, dst0_l); + ILVRL_H2_SW(zero, dst0, dst0_r, dst0_l); DOTP_SH2_SW(dst0_r, dst0_l, weight_vec, weight_vec, dst0_r, dst0_l); SRAR_W2_SW(dst0_r, dst0_l, rnd_vec); + dst0_r += offset_vec; + dst0_l += offset_vec; + CLIP_SW2_0_255(dst0_r, dst0_l); dst0 = __msa_pckev_h((v8i16) dst0_l, (v8i16) dst0_r); - dst0 += offset_vec; - CLIP_SH_0_255(dst0); out0 = (v16u8) __msa_pckev_b((v16i8) dst0, (v16i8) dst0); ST_W2(out0, 0, 1, dst, dst_stride); } else if (4 == height) { @@ -131,6 +148,7 @@ static void hevc_uniwgt_copy_6w_msa(const uint8_t *src, int32_t rnd_val) { uint32_t loop_cnt; + int32_t res = height & 0x07; uint64_t tp0, tp1, tp2, tp3; v16i8 zero = { 0 }; v16u8 out0, out1, out2, out3; @@ -138,9 +156,8 @@ static void hevc_uniwgt_copy_6w_msa(const uint8_t *src, v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, offset_vec; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = (height >> 3); loop_cnt--;) { @@ -181,6 +198,51 @@ static void hevc_uniwgt_copy_6w_msa(const uint8_t *src, ST_H2(out3, 2, 6, dst + 2 * dst_stride + 4, dst_stride); dst += (4 * dst_stride); } + if (res) { + LD4(src, src_stride, tp0, tp1, tp2, tp3); + src += (4 * src_stride); + INSERT_D2_SB(tp0, tp1, src0); + INSERT_D2_SB(tp2, tp3, src1); + LD4(src, src_stride, tp0, tp1, tp2, tp3); + src += (4 * src_stride); + INSERT_D2_SB(tp0, tp1, src2); + INSERT_D2_SB(tp2, tp3, src3); + + ILVRL_B2_SH(zero, src0, dst0, dst1); + ILVRL_B2_SH(zero, src1, dst2, dst3); + ILVRL_B2_SH(zero, src2, dst4, dst5); + ILVRL_B2_SH(zero, src3, dst6, dst7); + + SLLI_4V(dst0, dst1, dst2, dst3, 6); + SLLI_4V(dst4, dst5, dst6, dst7, 6); + + HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, + offset_vec, rnd_vec, dst0, dst1, dst2, + dst3); + HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, + offset_vec, rnd_vec, dst4, dst5, dst6, + dst7); + PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + + if (res == 2) { + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + } else if (res == 4) { + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); + ST_H2(out1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); + } else { + ST_W2(out0, 0, 2, dst, dst_stride); + ST_H2(out0, 2, 6, dst + 4, dst_stride); + ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); + ST_H2(out1, 2, 6, dst + 2 * dst_stride + 4, dst_stride); + dst += (4 * dst_stride); + ST_W2(out2, 0, 2, dst, dst_stride); + ST_H2(out2, 2, 6, dst + 4, dst_stride); + } + } } static void hevc_uniwgt_copy_8w_msa(const uint8_t *src, @@ -200,9 +262,8 @@ static void hevc_uniwgt_copy_8w_msa(const uint8_t *src, v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, offset_vec; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); if (2 == height) { @@ -295,9 +356,8 @@ static void hevc_uniwgt_copy_12w_msa(const uint8_t *src, v16i8 zero = { 0 }; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = 4; loop_cnt--;) { @@ -339,9 +399,8 @@ static void hevc_uniwgt_copy_16w_msa(const uint8_t *src, v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, offset_vec; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = height >> 2; loop_cnt--;) { @@ -383,9 +442,8 @@ static void hevc_uniwgt_copy_24w_msa(const uint8_t *src, v8i16 dst8, dst9, dst10, dst11; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = (height >> 2); loop_cnt--;) { @@ -435,9 +493,8 @@ static void hevc_uniwgt_copy_32w_msa(const uint8_t *src, v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, offset_vec; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = (height >> 1); loop_cnt--;) { @@ -482,9 +539,8 @@ static void hevc_uniwgt_copy_48w_msa(const uint8_t *src, v8i16 dst6, dst7, dst8, dst9, dst10, dst11; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = (height >> 1); loop_cnt--;) { @@ -539,9 +595,8 @@ static void hevc_uniwgt_copy_64w_msa(const uint8_t *src, v8i16 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; v4i32 weight_vec, rnd_vec; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); - offset_vec = __msa_fill_h(offset); + offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); for (loop_cnt = (height >> 1); loop_cnt--;) { @@ -596,33 +651,26 @@ static void hevc_hz_uniwgt_8t_4w_msa(const uint8_t *src, int32_t rnd_val) { uint32_t loop_cnt; + uint32_t res = height & 0x07; v16u8 out0, out1; - v8i16 filt0, filt1, filt2, filt3; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9, vec10; - v16i8 mask0, mask1, mask2, mask3, vec11, vec12, vec13, vec14, vec15; - v8i16 filter_vec, dst01, dst23, dst45, dst67; - v8i16 dst0, dst1, dst2, dst3, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v8i16 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; + v16i8 mask0, mask1, mask2, mask3; + v8i16 filter_vec, filt0, filt1, filt2, filt3; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[16]); mask1 = mask0 + 2; @@ -632,8 +680,6 @@ static void hevc_hz_uniwgt_8t_4w_msa(const uint8_t *src, for (loop_cnt = (height >> 3); loop_cnt--;) { LD_SB8(src, src_stride, src0, src1, src2, src3, src4, src5, src6, src7); src += (8 * src_stride); - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); - VSHF_B4_SB(src0, src1, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); VSHF_B4_SB(src2, src3, mask0, mask1, mask2, mask3, @@ -642,23 +688,78 @@ static void hevc_hz_uniwgt_8t_4w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src6, src7, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst01 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst23 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst45 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst67 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst01, dst23, dst45, dst67, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); } + if (res) { + LD_SB8(src, src_stride, src0, src1, src2, src3, src4, src5, src6, src7); + VSHF_B4_SB(src0, src1, mask0, mask1, mask2, mask3, + vec0, vec1, vec2, vec3); + VSHF_B4_SB(src2, src3, mask0, mask1, mask2, mask3, + vec4, vec5, vec6, vec7); + VSHF_B4_SB(src4, src5, mask0, mask1, mask2, mask3, + vec8, vec9, vec10, vec11); + VSHF_B4_SB(src6, src7, mask0, mask1, mask2, mask3, + vec12, vec13, vec14, vec15); + + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + + if (res == 2) { + ST_W2(out0, 0, 1, dst, dst_stride); + } else if (res == 4) { + ST_W4(out0, 0, 1, 2, 3, dst, dst_stride); + } else { + ST_W4(out0, 0, 1, 2, 3, dst, dst_stride); + ST_W2(out1, 0, 1, dst + 4 * dst_stride, dst_stride); + } + } } static void hevc_hz_uniwgt_8t_8w_msa(const uint8_t *src, @@ -672,35 +773,27 @@ static void hevc_hz_uniwgt_8t_8w_msa(const uint8_t *src, int32_t rnd_val) { uint32_t loop_cnt; + uint32_t res = height & 0x03; v16u8 out0, out1; v16i8 src0, src1, src2, src3; v8i16 filt0, filt1, filt2, filt3; v16i8 mask0, mask1, mask2, mask3; v8i16 filter_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3; - v8i16 weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v8i16 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -710,7 +803,6 @@ static void hevc_hz_uniwgt_8t_8w_msa(const uint8_t *src, for (loop_cnt = (height >> 2); loop_cnt--;) { LD_SB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); - XORI_B4_128_SB(src0, src1, src2, src3); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -720,23 +812,54 @@ static void hevc_hz_uniwgt_8t_8w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); dst += (4 * dst_stride); } + if (res) { + LD_SB2(src, src_stride, src0, src1); + + VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, + vec0, vec1, vec2, vec3); + VSHF_B4_SB(src1, src1, mask0, mask1, mask2, mask3, + vec4, vec5, vec6, vec7); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, vec0, vec1); + out0 = __msa_pckev_b((v16i8) vec1, (v16i8) vec0); + ST_D2(out0, 0, 1, dst, dst_stride); + } } static void hevc_hz_uniwgt_8t_12w_msa(const uint8_t *src, @@ -750,35 +873,26 @@ static void hevc_hz_uniwgt_8t_12w_msa(const uint8_t *src, int32_t rnd_val) { uint32_t loop_cnt; - v16u8 out0, out1, out2; + v16u8 out0, out1; v8i16 filt0, filt1, filt2, filt3; - v16i8 src0, src1, src2, src3, src4, src5, src6, src7; + v16i8 src0, src1, src2, src3; v16i8 mask0, mask1, mask2, mask3, mask4, mask5, mask6, mask7; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; v8i16 filter_vec; - v8i16 dst01, dst23, dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3; + v4i32 dst00, dst01; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -789,47 +903,41 @@ static void hevc_hz_uniwgt_8t_12w_msa(const uint8_t *src, mask6 = mask4 + 4; mask7 = mask4 + 6; - for (loop_cnt = (height >> 2); loop_cnt--;) { - LD_SB4(src, src_stride, src0, src1, src2, src3); - LD_SB4(src + 8, src_stride, src4, src5, src6, src7); - src += (4 * src_stride); - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); + for (loop_cnt = (height >> 1); loop_cnt--;) { + LD_SB2(src, src_stride, src0, src1); + LD_SB2(src + 8, src_stride, src2, src3); + src += (2 * src_stride); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); VSHF_B4_SB(src1, src1, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); - VSHF_B4_SB(src2, src2, mask0, mask1, mask2, mask3, - vec8, vec9, vec10, vec11); - VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, - vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); - VSHF_B4_SB(src4, src5, mask4, mask5, mask6, mask7, + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + VSHF_B4_SB(src2, src3, mask4, mask5, mask6, mask7, vec0, vec1, vec2, vec3); - VSHF_B4_SB(src6, src7, mask4, mask5, mask6, mask7, - vec4, vec5, vec6, vec7); - dst01 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst23 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst00, dst01); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst01, dst23, weight_vec, offset_vec, - rnd_vec, dst4, dst5); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL2(dst00, weight_vec, dst01, weight_vec, dst00, dst01); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W2_SW(dst00, dst01, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD2(dst00, offset_vec, dst01, offset_vec, dst00, dst01); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + CLIP_SW2_0_255(dst00, dst01); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, vec0, vec1); + vec2 = __msa_pckev_h((v8i16) dst01, (v8i16) dst00); + PCKEV_B2_UB(vec1, vec0, zero, vec2, out0, out1); - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); - ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); - ST_W4(out2, 0, 1, 2, 3, dst + 8, dst_stride); - dst += (4 * dst_stride); + ST_D2(out0, 0, 1, dst, dst_stride); + ST_W2(out1, 0, 1, dst + 8, dst_stride); + dst += (2 * dst_stride); } } @@ -851,27 +959,19 @@ static void hevc_hz_uniwgt_8t_16w_msa(const uint8_t *src, v8i16 filter_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3; - v8i16 weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -882,7 +982,6 @@ static void hevc_hz_uniwgt_8t_16w_msa(const uint8_t *src, LD_SB2(src, src_stride, src0, src2); LD_SB2(src + 8, src_stride, src1, src3); src += (2 * src_stride); - XORI_B4_128_SB(src0, src1, src2, src3); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -892,20 +991,30 @@ static void hevc_hz_uniwgt_8t_16w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); ST_UB2(out0, out1, dst, dst_stride); dst += (2 * dst_stride); } @@ -928,27 +1037,21 @@ static void hevc_hz_uniwgt_8t_24w_msa(const uint8_t *src, v16i8 mask0, mask1, mask2, mask3, mask4, mask5, mask6, mask7; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11; + v8i16 filter_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -964,7 +1067,6 @@ static void hevc_hz_uniwgt_8t_24w_msa(const uint8_t *src, src += src_stride; LD_SB2(src, 16, src2, src3); src += src_stride; - XORI_B4_128_SB(src0, src1, src2, src3); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); VSHF_B4_SB(src0, src1, mask4, mask5, mask6, mask7, @@ -973,31 +1075,46 @@ static void hevc_hz_uniwgt_8t_24w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src2, src2, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); VSHF_B4_SB(src2, src3, mask4, mask5, mask6, mask7, vec0, vec1, vec2, vec3); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); - dst4 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst5 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, - rnd_vec, dst4, dst5); - - PCKEV_B3_UB(dst1, dst0, dst4, dst3, dst5, dst2, out0, out1, out2); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst8, dst9); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst10, dst11); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11) + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, vec4, vec5); + + PCKEV_B3_UB(vec1, vec0, vec4, vec3, vec5, vec2, out0, out1, out2); ST_UB2(out0, out1, dst, dst_stride); ST_D2(out2, 0, 1, dst + 16, dst_stride); dst += (2 * dst_stride); @@ -1022,27 +1139,20 @@ static void hevc_hz_uniwgt_8t_32w_msa(const uint8_t *src, v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; v8i16 filter_vec; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst10, dst11, dst12, dst13, dst14, dst15, dst16, dst17; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -1054,7 +1164,6 @@ static void hevc_hz_uniwgt_8t_32w_msa(const uint8_t *src, src += src_stride; LD_SB4(src, 8, src4, src5, src6, src7); src += src_stride; - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1064,14 +1173,14 @@ static void hevc_hz_uniwgt_8t_32w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); VSHF_B4_SB(src4, src4, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1081,24 +1190,44 @@ static void hevc_hz_uniwgt_8t_32w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src7, src7, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst4 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst5 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst6 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst7 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, - offset_vec, rnd_vec, dst4, dst5, dst6, - dst7); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst10, dst11); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst12, dst13); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst14, dst15); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst16, dst17); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + MUL4(dst10, weight_vec, dst11, weight_vec, dst12, weight_vec, dst13, + weight_vec, dst10, dst11, dst12, dst13) + MUL4(dst14, weight_vec, dst15, weight_vec, dst16, weight_vec, dst17, + weight_vec, dst14, dst15, dst16, dst17); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + SRAR_W4_SW(dst10, dst11, dst12, dst13, rnd_vec); + SRAR_W4_SW(dst14, dst15, dst16, dst17, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + ADD4(dst10, offset_vec, dst11, offset_vec, dst12, offset_vec, dst13, + offset_vec, dst10, dst11, dst12, dst13); + ADD4(dst14, offset_vec, dst15, offset_vec, dst16, offset_vec, dst17, + offset_vec, dst14, dst15, dst16, dst17); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst10, dst11, dst12, dst13, dst14, dst15, dst16, dst17); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_H4_SH(dst11, dst10, dst13, dst12, dst15, dst14, dst17, dst16, + vec4, vec5, vec6, vec7); + + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + PCKEV_B2_UB(vec5, vec4, vec7, vec6, out2, out3); ST_UB2(out0, out1, dst, 16); dst += dst_stride; ST_UB2(out2, out3, dst, 16); @@ -1123,28 +1252,21 @@ static void hevc_hz_uniwgt_8t_48w_msa(const uint8_t *src, v16i8 mask0, mask1, mask2, mask3, mask4, mask5, mask6, mask7; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11; + v8i16 filter_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -1159,7 +1281,6 @@ static void hevc_hz_uniwgt_8t_48w_msa(const uint8_t *src, LD_SB3(src, 16, src0, src1, src2); src3 = LD_SB(src + 40); src += src_stride; - XORI_B4_128_SB(src0, src1, src2, src3); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1169,31 +1290,46 @@ static void hevc_hz_uniwgt_8t_48w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src1, src2, mask4, mask5, mask6, mask7, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); VSHF_B4_SB(src2, src2, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); - dst4 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst5 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, - rnd_vec, dst4, dst5); - - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst8, dst9); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst10, dst11); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11) + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, vec4, vec5); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + out2 = __msa_pckev_b((v16i8) vec5, (v16i8) vec4); ST_UB2(out0, out1, dst, 16); ST_UB(out2, dst + 32); dst += dst_stride; @@ -1219,27 +1355,20 @@ static void hevc_hz_uniwgt_8t_64w_msa(const uint8_t *src, v16i8 mask0, mask1, mask2, mask3, mask4, mask5, mask6, mask7; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 filter_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 3; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -1258,7 +1387,6 @@ static void hevc_hz_uniwgt_8t_64w_msa(const uint8_t *src, LD_SB2(src_tmp, 16, src0, src1); src2 = LD_SB(src_tmp + 24); src_tmp += 32; - XORI_B3_128_SB(src0, src1, src2); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1268,20 +1396,28 @@ static void hevc_hz_uniwgt_8t_64w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src2, src2, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, - filt2, filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, - filt2, filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, - filt2, filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, - dst2, dst3); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst6, dst7); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); ST_UB2(out0, out1, dst_tmp, 16); dst_tmp += 32; } @@ -1302,6 +1438,7 @@ static void hevc_vt_uniwgt_8t_4w_msa(const uint8_t *src, int32_t rnd_val) { int32_t loop_cnt; + int32_t res = height & 0x07; v16u8 out0, out1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; v16i8 src9, src10, src11, src12, src13, src14; @@ -1310,29 +1447,23 @@ static void hevc_vt_uniwgt_8t_4w_msa(const uint8_t *src, v16i8 src1110_r, src1211_r, src1312_r, src1413_r; v16i8 src2110, src4332, src6554, src8776, src10998; v16i8 src12111110, src14131312; - v8i16 filter_vec, dst01, dst23, dst45, dst67; + v8i16 filter_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; v8i16 filt0, filt1, filt2, filt3; - v8i16 dst0, dst1, dst2, dst3, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 vec0, vec1, vec2, vec3; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (3 * src_stride); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); src += (7 * src_stride); @@ -1345,7 +1476,6 @@ static void hevc_vt_uniwgt_8t_4w_msa(const uint8_t *src, ILVR_D3_SB(src21_r, src10_r, src43_r, src32_r, src65_r, src54_r, src2110, src4332, src6554); - XORI_B3_128_SB(src2110, src4332, src6554); for (loop_cnt = (height >> 3); loop_cnt--;) { LD_SB8(src, src_stride, @@ -1358,21 +1488,30 @@ static void hevc_vt_uniwgt_8t_4w_msa(const uint8_t *src, ILVR_D4_SB(src87_r, src76_r, src109_r, src98_r, src1211_r, src1110_r, src1413_r, src1312_r, src8776, src10998, src12111110, src14131312); - XORI_B4_128_SB(src8776, src10998, src12111110, src14131312); - dst01 = HEVC_FILT_8TAP_SH(src2110, src4332, src6554, src8776, filt0, - filt1, filt2, filt3); - dst23 = HEVC_FILT_8TAP_SH(src4332, src6554, src8776, src10998, filt0, - filt1, filt2, filt3); - dst45 = HEVC_FILT_8TAP_SH(src6554, src8776, src10998, src12111110, - filt0, filt1, filt2, filt3); - dst67 = HEVC_FILT_8TAP_SH(src8776, src10998, src12111110, src14131312, - filt0, filt1, filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst01, dst23, dst45, dst67, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); + HEVC_FILT_8TAP_4W_SH(src2110, src4332, src6554, src8776, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src4332, src6554, src8776, src10998, filt0, + filt1, filt2, filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(src6554, src8776, src10998, src12111110, + filt0, filt1, filt2, filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(src8776, src10998, src12111110, src14131312, + filt0, filt1, filt2, filt3, dst6, dst7); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); @@ -1381,6 +1520,48 @@ static void hevc_vt_uniwgt_8t_4w_msa(const uint8_t *src, src6554 = src14131312; src6 = src14; } + if (res) { + LD_SB8(src, src_stride, + src7, src8, src9, src10, src11, src12, src13, src14); + ILVR_B4_SB(src7, src6, src8, src7, src9, src8, src10, src9, + src76_r, src87_r, src98_r, src109_r); + ILVR_B4_SB(src11, src10, src12, src11, src13, src12, src14, src13, + src1110_r, src1211_r, src1312_r, src1413_r); + ILVR_D4_SB(src87_r, src76_r, src109_r, src98_r, src1211_r, src1110_r, + src1413_r, src1312_r, + src8776, src10998, src12111110, src14131312); + HEVC_FILT_8TAP_4W_SH(src2110, src4332, src6554, src8776, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src4332, src6554, src8776, src10998, filt0, + filt1, filt2, filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(src6554, src8776, src10998, src12111110, + filt0, filt1, filt2, filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(src8776, src10998, src12111110, src14131312, + filt0, filt1, filt2, filt3, dst6, dst7); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + + if (res == 2) { + ST_W2(out0, 0, 1, dst, dst_stride); + } else if (res == 4) { + ST_W4(out0, 0, 1, 2, 3, dst, dst_stride); + } else { + ST_W4(out0, 0, 1, 2, 3, dst, dst_stride); + ST_W2(out1, 0, 1, dst + 4 * dst_stride, dst_stride); + } + } } static void hevc_vt_uniwgt_8t_8w_msa(const uint8_t *src, @@ -1394,36 +1575,29 @@ static void hevc_vt_uniwgt_8t_8w_msa(const uint8_t *src, int32_t rnd_val) { int32_t loop_cnt; + int32_t res = height & 0x03; v16u8 out0, out1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 src10_r, src32_r, src54_r, src76_r, src98_r; v16i8 src21_r, src43_r, src65_r, src87_r, src109_r; v8i16 filt0, filt1, filt2, filt3; - v8i16 filter_vec; - v8i16 dst0, dst1, dst2, dst3, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, vec0, vec1, vec2, vec3; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (3 * src_stride); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); src += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); ILVR_B4_SB(src1, src0, src3, src2, src5, src4, src2, src1, src10_r, src32_r, src54_r, src21_r); @@ -1432,23 +1606,30 @@ static void hevc_vt_uniwgt_8t_8w_msa(const uint8_t *src, for (loop_cnt = (height >> 2); loop_cnt--;) { LD_SB4(src, src_stride, src7, src8, src9, src10); src += (4 * src_stride); - XORI_B4_128_SB(src7, src8, src9, src10); ILVR_B4_SB(src7, src6, src8, src7, src9, src8, src10, src9, src76_r, src87_r, src98_r, src109_r); - dst0 = HEVC_FILT_8TAP_SH(src10_r, src32_r, src54_r, src76_r, filt0, - filt1, filt2, filt3); - dst1 = HEVC_FILT_8TAP_SH(src21_r, src43_r, src65_r, src87_r, filt0, - filt1, filt2, filt3); - dst2 = HEVC_FILT_8TAP_SH(src32_r, src54_r, src76_r, src98_r, filt0, - filt1, filt2, filt3); - dst3 = HEVC_FILT_8TAP_SH(src43_r, src65_r, src87_r, src109_r, filt0, - filt1, filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + HEVC_FILT_8TAP_4W_SH(src10_r, src32_r, src54_r, src76_r, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src21_r, src43_r, src65_r, src87_r, filt0, + filt1, filt2, filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(src32_r, src54_r, src76_r, src98_r, + filt0, filt1, filt2, filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(src43_r, src65_r, src87_r, src109_r, + filt0, filt1, filt2, filt3, dst6, dst7); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); dst += (4 * dst_stride); @@ -1460,6 +1641,23 @@ static void hevc_vt_uniwgt_8t_8w_msa(const uint8_t *src, src65_r = src109_r; src6 = src10; } + if (res) { + LD_SB2(src, src_stride, src7, src8); + ILVR_B2_SB(src7, src6, src8, src7, src76_r, src87_r); + HEVC_FILT_8TAP_4W_SH(src10_r, src32_r, src54_r, src76_r, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src21_r, src43_r, src65_r, src87_r, filt0, + filt1, filt2, filt3, dst2, dst3); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, vec0, vec1); + out0 = __msa_pckev_b((v16i8) vec1, (v16i8) vec0); + ST_D2(out0, 0, 1, dst, dst_stride); + } } static void hevc_vt_uniwgt_8t_12w_msa(const uint8_t *src, @@ -1481,32 +1679,24 @@ static void hevc_vt_uniwgt_8t_12w_msa(const uint8_t *src, v16i8 src21_l, src43_l, src65_l, src87_l, src109_l; v16i8 src2110, src4332, src6554, src8776, src10998; v8i16 filt0, filt1, filt2, filt3; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 weight_vec_h, offset_vec, denom_vec, filter_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5; + v4i32 dst6, dst7, dst8, dst9, dst10, dst11; + v8i16 filter_vec, vec0, vec1, vec2, vec3, vec4, vec5; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (3 * src_stride); - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); src += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); ILVR_B4_SB(src1, src0, src3, src2, src5, src4, src2, src1, src10_r, src32_r, src54_r, src21_r); @@ -1520,7 +1710,6 @@ static void hevc_vt_uniwgt_8t_12w_msa(const uint8_t *src, for (loop_cnt = 4; loop_cnt--;) { LD_SB4(src, src_stride, src7, src8, src9, src10); src += (4 * src_stride); - XORI_B4_128_SB(src7, src8, src9, src10); ILVR_B4_SB(src7, src6, src8, src7, src9, src8, src10, src9, src76_r, src87_r, src98_r, src109_r); @@ -1528,26 +1717,40 @@ static void hevc_vt_uniwgt_8t_12w_msa(const uint8_t *src, src76_l, src87_l, src98_l, src109_l); ILVR_D2_SB(src87_l, src76_l, src109_l, src98_l, src8776, src10998); - dst0 = HEVC_FILT_8TAP_SH(src10_r, src32_r, src54_r, src76_r, filt0, - filt1, filt2, filt3); - dst1 = HEVC_FILT_8TAP_SH(src21_r, src43_r, src65_r, src87_r, filt0, - filt1, filt2, filt3); - dst2 = HEVC_FILT_8TAP_SH(src32_r, src54_r, src76_r, src98_r, filt0, - filt1, filt2, filt3); - dst3 = HEVC_FILT_8TAP_SH(src43_r, src65_r, src87_r, src109_r, filt0, - filt1, filt2, filt3); - dst4 = HEVC_FILT_8TAP_SH(src2110, src4332, src6554, src8776, filt0, - filt1, filt2, filt3); - dst5 = HEVC_FILT_8TAP_SH(src4332, src6554, src8776, src10998, filt0, - filt1, filt2, filt3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, - rnd_vec, dst4, dst5); - - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); + HEVC_FILT_8TAP_4W_SH(src10_r, src32_r, src54_r, src76_r, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src21_r, src43_r, src65_r, src87_r, filt0, + filt1, filt2, filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(src32_r, src54_r, src76_r, src98_r, + filt0, filt1, filt2, filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(src43_r, src65_r, src87_r, src109_r, + filt0, filt1, filt2, filt3, dst6, dst7); + HEVC_FILT_8TAP_4W_SH(src2110, src4332, src6554, src8776, + filt0, filt1, filt2, filt3, dst8, dst9); + HEVC_FILT_8TAP_4W_SH(src4332, src6554, src8776, src10998, + filt0, filt1, filt2, filt3, dst10, dst11); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, vec4, vec5); + PCKEV_B3_UB(vec1, vec0, vec3, vec2, vec5, vec4, out0, out1, out2); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_W4(out2, 0, 1, 2, 3, dst + 8, dst_stride); dst += (4 * dst_stride); @@ -1579,6 +1782,7 @@ static void hevc_vt_uniwgt_8t_16multx4mult_msa(const uint8_t *src, const uint8_t *src_tmp; uint8_t *dst_tmp; int32_t loop_cnt, cnt; + int32_t res = height & 0x03; v16u8 out0, out1, out2, out3; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 src10_r, src32_r, src54_r, src76_r; @@ -1587,28 +1791,22 @@ static void hevc_vt_uniwgt_8t_16multx4mult_msa(const uint8_t *src, v16i8 src21_l, src43_l, src65_l, src87_l; v16i8 src98_r, src109_r, src98_l, src109_l; v8i16 filt0, filt1, filt2, filt3; - v8i16 filter_vec; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (3 * src_stride); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); + offset_vec = __msa_fill_w(offset); - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); filter_vec = LD_SH(filter); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); for (cnt = weightmul16; cnt--;) { src_tmp = src; @@ -1616,12 +1814,10 @@ static void hevc_vt_uniwgt_8t_16multx4mult_msa(const uint8_t *src, LD_SB7(src_tmp, src_stride, src0, src1, src2, src3, src4, src5, src6); src_tmp += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); for (loop_cnt = (height >> 2); loop_cnt--;) { LD_SB4(src_tmp, src_stride, src7, src8, src9, src10); src_tmp += (4 * src_stride); - XORI_B4_128_SB(src7, src8, src9, src10); ILVR_B4_SB(src1, src0, src3, src2, src5, src4, src2, src1, src10_r, src32_r, src54_r, src21_r); @@ -1634,31 +1830,52 @@ static void hevc_vt_uniwgt_8t_16multx4mult_msa(const uint8_t *src, ILVL_B4_SB(src7, src6, src8, src7, src9, src8, src10, src9, src76_l, src87_l, src98_l, src109_l); - dst0 = HEVC_FILT_8TAP_SH(src10_r, src32_r, src54_r, src76_r, filt0, - filt1, filt2, filt3); - dst1 = HEVC_FILT_8TAP_SH(src10_l, src32_l, src54_l, src76_l, filt0, - filt1, filt2, filt3); - dst2 = HEVC_FILT_8TAP_SH(src21_r, src43_r, src65_r, src87_r, filt0, - filt1, filt2, filt3); - dst3 = HEVC_FILT_8TAP_SH(src21_l, src43_l, src65_l, src87_l, filt0, - filt1, filt2, filt3); - dst4 = HEVC_FILT_8TAP_SH(src32_r, src54_r, src76_r, src98_r, filt0, - filt1, filt2, filt3); - dst5 = HEVC_FILT_8TAP_SH(src32_l, src54_l, src76_l, src98_l, filt0, - filt1, filt2, filt3); - dst6 = HEVC_FILT_8TAP_SH(src43_r, src65_r, src87_r, src109_r, filt0, - filt1, filt2, filt3); - dst7 = HEVC_FILT_8TAP_SH(src43_l, src65_l, src87_l, src109_l, filt0, - filt1, filt2, filt3); + HEVC_FILT_8TAP_4W_SH(src10_r, src32_r, src54_r, src76_r, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src10_l, src32_l, src54_l, src76_l, filt0, + filt1, filt2, filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(src21_r, src43_r, src65_r, src87_r, filt0, + filt1, filt2, filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(src21_l, src43_l, src65_l, src87_l, filt0, + filt1, filt2, filt3, dst6, dst7); + HEVC_FILT_8TAP_4W_SH(src32_r, src54_r, src76_r, src98_r, filt0, + filt1, filt2, filt3, dst8, dst9); + HEVC_FILT_8TAP_4W_SH(src32_l, src54_l, src76_l, src98_l, filt0, + filt1, filt2, filt3, dst10, dst11); + HEVC_FILT_8TAP_4W_SH(src43_r, src65_r, src87_r, src109_r, filt0, + filt1, filt2, filt3, dst12, dst13); + HEVC_FILT_8TAP_4W_SH(src43_l, src65_l, src87_l, src109_l, filt0, + filt1, filt2, filt3, dst14, dst15); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, dst15, + weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, dst15, + offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_H4_SH(dst9, dst8, dst11, dst10, dst13, dst12, dst15, + dst14, vec4, vec5, vec6, vec7); + PCKEV_B4_UB(vec1, vec0, vec3, vec2, vec5, vec4, vec7, vec6, + out0, out1, out2, out3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, - dst2, dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, - offset_vec, rnd_vec, dst4, dst5, - dst6, dst7); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); ST_UB4(out0, out1, out2, out3, dst_tmp, dst_stride); dst_tmp += (4 * dst_stride); @@ -1670,6 +1887,43 @@ static void hevc_vt_uniwgt_8t_16multx4mult_msa(const uint8_t *src, src5 = src9; src6 = src10; } + if (res) { + LD_SB2(src_tmp, src_stride, src7, src8); + + ILVR_B4_SB(src1, src0, src3, src2, src5, src4, src2, src1, + src10_r, src32_r, src54_r, src21_r); + ILVR_B2_SB(src4, src3, src6, src5, src43_r, src65_r); + ILVL_B4_SB(src1, src0, src3, src2, src5, src4, src2, src1, + src10_l, src32_l, src54_l, src21_l); + ILVL_B2_SB(src4, src3, src6, src5, src43_l, src65_l); + ILVR_B2_SB(src7, src6, src8, src7, src76_r, src87_r); + ILVL_B2_SB(src7, src6, src8, src7, src76_l, src87_l); + + HEVC_FILT_8TAP_4W_SH(src10_r, src32_r, src54_r, src76_r, filt0, + filt1, filt2, filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(src10_l, src32_l, src54_l, src76_l, filt0, + filt1, filt2, filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(src21_r, src43_r, src65_r, src87_r, filt0, + filt1, filt2, filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(src21_l, src43_l, src65_l, src87_l, filt0, + filt1, filt2, filt3, dst6, dst7); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3) + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, dst7, + weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + vec0, vec1, vec2, vec3); + PCKEV_B2_UB(vec1, vec0, vec3, vec2, out0, out1); + + ST_UB2(out0, out1, dst_tmp, dst_stride); + } src += 16; dst += 16; @@ -1774,20 +2028,21 @@ static void hevc_hv_uniwgt_8t_4w_msa(const uint8_t *src, v8i16 filter_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst30, dst41, dst52, dst63, dst66, dst97, dst108; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8, dst9, dst10; v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst98_r; v8i16 dst21_r, dst43_r, dst65_r, dst87_r, dst109_r; v4i32 dst0_r, dst1_r, dst2_r, dst3_r; - v4i32 weight_vec, offset_vec, rnd_vec, const_128, denom_vec; + v4i32 weight_vec, offset_vec, rnd_vec; v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); + v8i16 zero = { 0 }; src -= ((3 * src_stride) + 3); filter_vec = LD_SH(filter_x); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W4_SH(filter_vec, filt_h0, filt_h1, filt_h2, filt_h3); mask1 = mask0 + 2; @@ -1797,15 +2052,9 @@ static void hevc_hv_uniwgt_8t_4w_msa(const uint8_t *src, weight_vec = __msa_fill_w(weight); offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); - denom_vec = rnd_vec - 6; - - const_128 = __msa_ldi_w(128); - const_128 *= weight_vec; - offset_vec += __msa_srar_w(const_128, denom_vec); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); src += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); /* row 0 row 1 row 2 row 3 */ VSHF_B4_SB(src0, src3, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1814,39 +2063,41 @@ static void hevc_hv_uniwgt_8t_4w_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src3, src6, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst30 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst41 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst52 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst63 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, - filt3); - - ILVRL_H2_SH(dst41, dst30, dst10_r, dst43_r); - ILVRL_H2_SH(dst52, dst41, dst21_r, dst54_r); - ILVRL_H2_SH(dst63, dst52, dst32_r, dst65_r); - - dst66 = (v8i16) __msa_splati_d((v2i64) dst63, 1); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst3); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst1, dst4); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst2, dst5); + vec0 = __msa_ilvl_b((v16i8) zero, (v16i8) vec12); + vec1 = __msa_ilvl_b((v16i8) zero, (v16i8) vec13); + vec2 = __msa_ilvl_b((v16i8) zero, (v16i8) vec14); + vec3 = __msa_ilvl_b((v16i8) zero, (v16i8) vec15); + dst6 = __msa_dotp_s_w((v8i16) vec0, (v8i16) filt0); + dst6 = __msa_dpadd_s_w((v4i32) dst6, (v8i16) vec1, (v8i16) filt1); + dst6 = __msa_dpadd_s_w((v4i32) dst6, (v8i16) vec2, (v8i16) filt2); + dst6 = __msa_dpadd_s_w((v4i32) dst6, (v8i16) vec3, (v8i16) filt3); + + ILVEV_H2_SH(dst0, dst1, dst3, dst4, dst10_r, dst43_r); + ILVEV_H2_SH(dst1, dst2, dst4, dst5, dst21_r, dst54_r); + ILVEV_H2_SH(dst2, dst3, dst5, dst6, dst32_r, dst65_r); for (loop_cnt = height >> 2; loop_cnt--;) { LD_SB4(src, src_stride, src7, src8, src9, src10); src += (4 * src_stride); - XORI_B4_128_SB(src7, src8, src9, src10); VSHF_B4_SB(src7, src9, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); VSHF_B4_SB(src8, src10, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); - dst97 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst108 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst7, dst9); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst8, dst10); - dst76_r = __msa_ilvr_h(dst97, dst66); - ILVRL_H2_SH(dst108, dst97, dst87_r, dst109_r); - dst66 = (v8i16) __msa_splati_d((v2i64) dst97, 1); - dst98_r = __msa_ilvr_h(dst66, dst108); + dst76_r = __msa_ilvev_h((v8i16) dst7, (v8i16) dst6); + ILVEV_H2_SH(dst7, dst8, dst9, dst10, dst87_r, dst109_r); + dst98_r = __msa_ilvev_h((v8i16) dst9, (v8i16) dst8); dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, filt_h1, filt_h2, filt_h3); @@ -1875,7 +2126,7 @@ static void hevc_hv_uniwgt_8t_4w_msa(const uint8_t *src, dst21_r = dst65_r; dst43_r = dst87_r; dst65_r = dst109_r; - dst66 = (v8i16) __msa_splati_d((v2i64) dst108, 1); + dst6 = dst10; } } @@ -1896,33 +2147,32 @@ static void hevc_hv_uniwgt_8t_8multx2mult_msa(const uint8_t *src, uint8_t *dst_tmp; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; v8i16 filt0, filt1, filt2, filt3; - v4i32 filt_h0, filt_h1, filt_h2, filt_h3; + v8i16 filt_h0, filt_h1, filt_h2, filt_h3; v16i8 mask1, mask2, mask3; v8i16 filter_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8; v4i32 dst0_r, dst0_l, dst1_r, dst1_l; v8i16 dst10_r, dst32_r, dst54_r, dst76_r; v8i16 dst10_l, dst32_l, dst54_l, dst76_l; v8i16 dst21_r, dst43_r, dst65_r, dst87_r; v8i16 dst21_l, dst43_l, dst65_l, dst87_l; - v4i32 weight_vec, offset_vec, rnd_vec, const_128, denom_vec; + v4i32 weight_vec, offset_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8, dst9, dst10; + v4i32 dst11, dst12, dst13, dst14, dst15; v16i8 mask0 = LD_SB(ff_hevc_mask_arr); + v8i16 zero = { 0 }; src -= ((3 * src_stride) + 3); weight_vec = __msa_fill_w(weight); offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); - denom_vec = rnd_vec - 6; - const_128 = __msa_ldi_w(128); - const_128 *= weight_vec; - offset_vec += __msa_srar_w(const_128, denom_vec); filter_vec = LD_SH(filter_x); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SW(filter_vec, filt0, filt1, filt2, filt3); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); @@ -1938,7 +2188,6 @@ static void hevc_hv_uniwgt_8t_8multx2mult_msa(const uint8_t *src, LD_SB7(src_tmp, src_stride, src0, src1, src2, src3, src4, src5, src6); src_tmp += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1948,14 +2197,14 @@ static void hevc_hv_uniwgt_8t_8multx2mult_msa(const uint8_t *src, vec8, vec9, vec10, vec11); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, + filt2, filt3, dst6, dst7); VSHF_B4_SB(src4, src4, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1963,31 +2212,30 @@ static void hevc_hv_uniwgt_8t_8multx2mult_msa(const uint8_t *src, vec4, vec5, vec6, vec7); VSHF_B4_SB(src6, src6, mask0, mask1, mask2, mask3, vec8, vec9, vec10, vec11); - dst4 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst5 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst6 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - - ILVR_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst2, dst1, - dst10_r, dst32_r, dst54_r, dst21_r); - ILVR_H2_SH(dst4, dst3, dst6, dst5, dst43_r, dst65_r); - ILVL_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst2, dst1, - dst10_l, dst32_l, dst54_l, dst21_l); - ILVL_H2_SH(dst4, dst3, dst6, dst5, dst43_l, dst65_l); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst8, dst9); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst10, dst11); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst12, dst13); + + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst21_r, dst21_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst32_r, dst32_l); + ILVEV_H2_SH(dst6, dst8, dst7, dst9, dst43_r, dst43_l); + ILVEV_H2_SH(dst8, dst10, dst9, dst11, dst54_r, dst54_l); + ILVEV_H2_SH(dst10, dst12, dst11, dst13, dst65_r, dst65_l); for (loop_cnt = height >> 1; loop_cnt--;) { LD_SB2(src_tmp, src_stride, src7, src8); src_tmp += 2 * src_stride; - XORI_B2_128_SB(src7, src8); VSHF_B4_SB(src7, src7, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); - dst7 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, + filt2, filt3, dst14, dst15); - ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); + ILVEV_H2_SH(dst12, dst14, dst13, dst15, dst76_r, dst76_l); dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, filt_h1, filt_h2, filt_h3); dst0_l = HEVC_FILT_8TAP(dst10_l, dst32_l, dst54_l, dst76_l, @@ -1998,10 +2246,10 @@ static void hevc_hv_uniwgt_8t_8multx2mult_msa(const uint8_t *src, /* row 8 */ VSHF_B4_SB(src8, src8, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); - dst8 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, + filt2, filt3, dst0, dst1); - ILVRL_H2_SH(dst8, dst7, dst87_r, dst87_l); + ILVEV_H2_SH(dst14, dst0, dst15, dst1, dst87_r, dst87_l); dst1_r = HEVC_FILT_8TAP(dst21_r, dst43_r, dst65_r, dst87_r, filt_h0, filt_h1, filt_h2, filt_h3); dst1_l = HEVC_FILT_8TAP(dst21_l, dst43_l, dst65_l, dst87_l, @@ -2033,7 +2281,8 @@ static void hevc_hv_uniwgt_8t_8multx2mult_msa(const uint8_t *src, dst21_l = dst43_l; dst43_l = dst65_l; dst65_l = dst87_l; - dst6 = dst8; + dst12 = dst0; + dst13 = dst1; } src += 8; @@ -2076,19 +2325,21 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, v16i8 mask0, mask1, mask2, mask3, mask4, mask5, mask6, mask7; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 dst30, dst41, dst52, dst63, dst66, dst97, dst108; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; v8i16 filt0, filt1, filt2, filt3, filt_h0, filt_h1, filt_h2, filt_h3; v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst10_l, dst32_l, dst54_l; v8i16 dst98_r, dst21_r, dst43_r, dst65_r, dst87_r, dst109_r; v8i16 dst76_l, filter_vec; v4i32 dst0_r, dst0_l, dst1_r, dst2_r, dst3_r; - v4i32 weight_vec, offset_vec, rnd_vec, const_128, denom_vec; + v4i32 weight_vec, offset_vec, rnd_vec; + v8i16 zero = { 0 }; src -= ((3 * src_stride) + 3); filter_vec = LD_SH(filter_x); - SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W4_SH(filter_vec, filt0, filt1, filt2, filt3); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); @@ -2098,11 +2349,6 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, weight_vec = __msa_fill_w(weight); offset_vec = __msa_fill_w(offset); rnd_vec = __msa_fill_w(rnd_val); - denom_vec = rnd_vec - 6; - - const_128 = __msa_ldi_w(128); - const_128 *= weight_vec; - offset_vec += __msa_srar_w(const_128, denom_vec); mask0 = LD_SB(ff_hevc_mask_arr); mask1 = mask0 + 2; @@ -2114,7 +2360,6 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, LD_SB7(src_tmp, src_stride, src0, src1, src2, src3, src4, src5, src6); src_tmp += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); /* row 0 row 1 row 2 row 3 */ VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -2123,38 +2368,37 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, vec11); VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec12, vec13, vec14, vec15); - dst0 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst1 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst2 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst3 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, - filt2, filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst1); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst2, dst3); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst4, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, + filt2, filt3, dst6, dst7); VSHF_B4_SB(src4, src4, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); VSHF_B4_SB(src5, src5, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); VSHF_B4_SB(src6, src6, mask0, mask1, mask2, mask3, vec8, vec9, vec10, vec11); - dst4 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst5 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst6 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst8, dst9); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst10, dst11); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst12, dst13); for (loop_cnt = 16; loop_cnt--;) { src7 = LD_SB(src_tmp); - src7 = (v16i8) __msa_xori_b((v16u8) src7, 128); src_tmp += src_stride; VSHF_B4_SB(src7, src7, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); - dst7 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); - ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst14, dst15); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst32_r, dst32_l); + ILVEV_H2_SH(dst8, dst10, dst9, dst11, dst54_r, dst54_l); + ILVEV_H2_SH(dst12, dst14, dst13, dst15, dst76_r, dst76_l); dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, filt_h1, filt_h2, filt_h3); @@ -2172,13 +2416,20 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, ST_D1(out, 0, dst_tmp); dst_tmp += dst_stride; - dst0 = dst1; - dst1 = dst2; - dst2 = dst3; - dst3 = dst4; - dst4 = dst5; - dst5 = dst6; - dst6 = dst7; + dst0 = dst2; + dst1 = dst3; + dst2 = dst4; + dst3 = dst5; + dst4 = dst6; + dst5 = dst7; + dst6 = dst8; + dst7 = dst9; + dst8 = dst10; + dst9 = dst11; + dst10 = dst12; + dst11 = dst13; + dst12 = dst14; + dst13 = dst15; } src += 8; @@ -2191,7 +2442,6 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); src += (7 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); VSHF_B4_SB(src0, src3, mask4, mask5, mask6, mask7, vec0, vec1, vec2, vec3); VSHF_B4_SB(src1, src4, mask4, mask5, mask6, mask7, vec4, vec5, vec6, vec7); @@ -2199,38 +2449,33 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, vec11); VSHF_B4_SB(src3, src6, mask4, mask5, mask6, mask7, vec12, vec13, vec14, vec15); - dst30 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst41 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); - dst52 = HEVC_FILT_8TAP_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, - filt3); - dst63 = HEVC_FILT_8TAP_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, - filt3); - ILVRL_H2_SH(dst41, dst30, dst10_r, dst43_r); - ILVRL_H2_SH(dst52, dst41, dst21_r, dst54_r); - ILVRL_H2_SH(dst63, dst52, dst32_r, dst65_r); - - dst66 = (v8i16) __msa_splati_d((v2i64) dst63, 1); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst0, dst3); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst1, dst4); + HEVC_FILT_8TAP_4W_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, + filt3, dst2, dst5); + HEVC_FILT_8TAP_4W_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, + filt3, dst3, dst6); + ILVEV_H2_SH(dst0, dst1, dst3, dst4, dst10_r, dst43_r); + ILVEV_H2_SH(dst1, dst2, dst4, dst5, dst21_r, dst54_r); + ILVEV_H2_SH(dst2, dst3, dst5, dst6, dst32_r, dst65_r); for (loop_cnt = 4; loop_cnt--;) { LD_SB4(src, src_stride, src7, src8, src9, src10); src += (4 * src_stride); - XORI_B4_128_SB(src7, src8, src9, src10); VSHF_B4_SB(src7, src9, mask4, mask5, mask6, mask7, vec0, vec1, vec2, vec3); VSHF_B4_SB(src8, src10, mask4, mask5, mask6, mask7, vec4, vec5, vec6, vec7); - dst97 = HEVC_FILT_8TAP_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, - filt3); - dst108 = HEVC_FILT_8TAP_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, - filt3); + HEVC_FILT_8TAP_4W_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, + filt3, dst7, dst9); + HEVC_FILT_8TAP_4W_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, + filt3, dst8, dst10); - dst76_r = __msa_ilvr_h(dst97, dst66); - ILVRL_H2_SH(dst108, dst97, dst87_r, dst109_r); - dst66 = (v8i16) __msa_splati_d((v2i64) dst97, 1); - dst98_r = __msa_ilvr_h(dst66, dst108); + ILVEV_H2_SH(dst6, dst7, dst7, dst8, dst76_r, dst87_r); + ILVEV_H2_SH(dst9, dst10, dst8, dst9, dst109_r, dst98_r); dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, filt_h1, filt_h2, filt_h3); @@ -2259,7 +2504,7 @@ static void hevc_hv_uniwgt_8t_12w_msa(const uint8_t *src, dst21_r = dst65_r; dst43_r = dst87_r; dst65_r = dst109_r; - dst66 = (v8i16) __msa_splati_d((v2i64) dst108, 1); + dst6 = dst10; } } @@ -2353,50 +2598,41 @@ static void hevc_hz_uniwgt_4t_4x2_msa(const uint8_t *src, int32_t rnd_val) { v16u8 out; - v8i16 filt0, filt1; + v8i16 filt0, filt1, filter_vec; v16i8 src0, src1, vec0, vec1; + v8i16 tmp0, tmp1, tmp2, tmp3; v16i8 mask1; - v8i16 dst0; - v4i32 dst0_r, dst0_l; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1; + v4i32 weight_vec, rnd_vec, offset_vec; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[16]); + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); mask1 = mask0 + 2; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); LD_SB2(src, src_stride, src0, src1); - XORI_B2_128_SB(src0, src1); VSHF_B2_SB(src0, src1, src0, src1, mask0, mask1, vec0, vec1); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); - ILVRL_H2_SW(dst0, dst0, dst0_r, dst0_l); - DOTP_SH2_SW(dst0_r, dst0_l, weight_vec, weight_vec, dst0_r, dst0_l); - SRAR_W2_SW(dst0_r, dst0_l, rnd_vec); - dst0 = __msa_pckev_h((v8i16) dst0_l, (v8i16) dst0_r); - dst0 = __msa_adds_s_h(dst0, offset_vec); - CLIP_SH_0_255(dst0); - out = (v16u8) __msa_pckev_b((v16i8) dst0, (v16i8) dst0); + MUL2(dst0, weight_vec, dst1, weight_vec, dst0, dst1); + SRAR_W2_SW(dst0, dst1, rnd_vec); + ADD2(dst0, offset_vec, dst1, offset_vec, dst0, dst1); + CLIP_SW2_0_255(dst0, dst1); + vec0 = __msa_pckev_h((v8i16) dst1, (v8i16) dst0); + out = (v16u8) __msa_pckev_b((v16i8) vec0, (v16i8) vec0); ST_W2(out, 0, 1, dst, dst_stride); dst += (4 * dst_stride); } @@ -2414,48 +2650,49 @@ static void hevc_hz_uniwgt_4t_4x4_msa(const uint8_t *src, v8i16 filt0, filt1; v16i8 src0, src1, src2, src3; v16i8 mask1, vec0, vec1, vec2, vec3; - v8i16 dst0, dst1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3; + v8i16 filter_vec; + v4i32 weight_vec, rnd_vec, offset_vec; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[16]); + v8i16 zero = { 0 }; src -= 1; /* rearranging filter */ filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - mask1 = mask0 + 2; - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); + mask1 = mask0 + 2; LD_SB4(src, src_stride, src0, src1, src2, src3); - XORI_B4_128_SB(src0, src1, src2, src3); VSHF_B2_SB(src0, src1, src0, src1, mask0, mask1, vec0, vec1); VSHF_B2_SB(src2, src3, src2, src3, mask0, mask1, vec2, vec3); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst0, dst1, weight_vec, offset_vec, rnd_vec, - dst0, dst1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + tmp0 = __msa_pckev_h((v8i16) dst1, (v8i16) dst0); + tmp1 = __msa_pckev_h((v8i16) dst3, (v8i16) dst2); - out = (v16u8) __msa_pckev_b((v16i8) dst1, (v16i8) dst0); + out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_W4(out, 0, 1, 2, 3, dst, dst_stride); - dst += (4 * dst_stride); } static void hevc_hz_uniwgt_4t_4x8multiple_msa(const uint8_t *src, @@ -2473,31 +2710,22 @@ static void hevc_hz_uniwgt_4t_4x8multiple_msa(const uint8_t *src, v8i16 filt0, filt1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v16i8 mask1, vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; v8i16 filter_vec; - v8i16 weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 weight_vec, rnd_vec, offset_vec; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[16]); + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; @@ -2505,22 +2733,43 @@ static void hevc_hz_uniwgt_4t_4x8multiple_msa(const uint8_t *src, LD_SB8(src, src_stride, src0, src1, src2, src3, src4, src5, src6, src7); src += (8 * src_stride); - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); - VSHF_B2_SB(src0, src1, src0, src1, mask0, mask1, vec0, vec1); VSHF_B2_SB(src2, src3, src2, src3, mask0, mask1, vec2, vec3); VSHF_B2_SB(src4, src5, src4, src5, mask0, mask1, vec4, vec5); VSHF_B2_SB(src6, src7, src6, src7, mask0, mask1, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + tmp0 = __msa_pckev_h((v8i16) dst1, (v8i16) dst0); + tmp1 = __msa_pckev_h((v8i16) dst3, (v8i16) dst2); + tmp2 = __msa_pckev_h((v8i16) dst5, (v8i16) dst4); + tmp3 = __msa_pckev_h((v8i16) dst7, (v8i16) dst6); + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); } @@ -2560,65 +2809,101 @@ static void hevc_hz_uniwgt_4t_6w_msa(const uint8_t *src, int32_t rnd_val) { v16u8 out0, out1, out2, out3; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; LD_SB8(src, src_stride, src0, src1, src2, src3, src4, src5, src6, src7); - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec2, vec3); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec4, vec5); VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec6, vec7); - dst4 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, - weight_vec, offset_vec, rnd_vec, - dst4, dst5, dst6, dst7); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst12 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); + PCKEV_B2_UB(tmp5, tmp4, tmp7, tmp6, out2, out3); ST_W2(out0, 0, 2, dst, dst_stride); ST_H2(out0, 2, 6, dst + 4, dst_stride); ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); @@ -2640,48 +2925,49 @@ static void hevc_hz_uniwgt_4t_8x2_msa(const uint8_t *src, int32_t rnd_val) { v16u8 out; - v8i16 filt0, filt1, dst0, dst1; + v8i16 filter_vec, filt0, filt1; v16i8 src0, src1; v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 weight_vec, rnd_vec, offset_vec; + v4i32 dst0, dst1, dst2, dst3; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); - + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; LD_SB2(src, src_stride, src0, src1); - XORI_B2_128_SB(src0, src1); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst0, dst1, weight_vec, offset_vec, rnd_vec, - dst0, dst1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); - out = (v16u8) __msa_pckev_b((v16i8) dst1, (v16i8) dst0); + out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_D2(out, 0, 1, dst, dst_stride); } @@ -2697,48 +2983,63 @@ static void hevc_hz_uniwgt_4t_8x4_msa(const uint8_t *src, v16u8 out0, out1; v16i8 src0, src1, src2, src3; v16i8 mask0, mask1, vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 filt0, filt1, dst0, dst1, dst2, dst3; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; LD_SB4(src, src_stride, src0, src1, src2, src3); - XORI_B4_128_SB(src0, src1, src2, src3); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); } @@ -2752,40 +3053,31 @@ static void hevc_hz_uniwgt_4t_8x6_msa(const uint8_t *src, int32_t rnd_val) { v16u8 out0, out1, out2; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16i8 src0, src1, src2, src3, src4, src5; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[0]); v16i8 mask1; v16i8 vec11; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9, vec10; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; LD_SB6(src, src_stride, src0, src1, src2, src3, src4, src5); - XORI_B6_128_SB(src0, src1, src2, src3, src4, src5); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); @@ -2793,21 +3085,55 @@ static void hevc_hz_uniwgt_4t_8x6_msa(const uint8_t *src, VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec8, vec9); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec10, vec11); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(vec8, vec9, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec10, vec11, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec8, tmp0, tmp1); + ILVRL_B2_SH(zero, vec9, tmp2, tmp3); + ILVRL_B2_SH(zero, vec10, tmp4, tmp5); + ILVRL_B2_SH(zero, vec11, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, rnd_vec, - dst4, dst5); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); + PCKEV_B3_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, out0, out1, out2); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_D2(out2, 0, 1, dst + 4 * dst_stride, dst_stride); } @@ -2823,70 +3149,107 @@ static void hevc_hz_uniwgt_4t_8x8multiple_msa(const uint8_t *src, int32_t rnd_val) { uint32_t loop_cnt; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16u8 out0, out1, out2, out3; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[0]); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; for (loop_cnt = (height >> 3); loop_cnt--;) { LD_SB8(src, src_stride, src0, src1, src2, src3, src4, src5, src6, src7); src += (8 * src_stride); - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec2, vec3); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec4, vec5); VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec6, vec7); - dst4 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, - weight_vec, offset_vec, rnd_vec, - dst4, dst5, dst6, dst7); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst12 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); + PCKEV_B2_UB(tmp5, tmp4, tmp7, tmp6, out2, out3); ST_D8(out0, out1, out2, out3, 0, 1, 0, 1, 0, 1, 0, 1, dst, dst_stride); dst += (8 * dst_stride); } @@ -2930,37 +3293,29 @@ static void hevc_hz_uniwgt_4t_12w_msa(const uint8_t *src, { uint32_t loop_cnt; v16u8 out0, out1, out2; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16i8 src0, src1, src2, src3; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[0]); v16i8 mask2 = { 8, 9, 9, 10, 10, 11, 11, 12, 24, 25, 25, 26, 26, 27, 27, 28 }; v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9, vec10; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; v16i8 mask3, vec11; - v4i32 weight_vec, rnd_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; mask3 = mask2 + 2; @@ -2969,29 +3324,62 @@ static void hevc_hz_uniwgt_4t_12w_msa(const uint8_t *src, LD_SB4(src, src_stride, src0, src1, src2, src3); src += (4 * src_stride); - XORI_B4_128_SB(src0, src1, src2, src3); - VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); VSHF_B2_SB(src0, src1, src0, src1, mask2, mask3, vec8, vec9); VSHF_B2_SB(src2, src3, src2, src3, mask2, mask3, vec10, vec11); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(vec8, vec9, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec10, vec11, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, - rnd_vec, dst4, dst5); - - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec8, tmp0, tmp1); + ILVRL_B2_SH(zero, vec9, tmp2, tmp3); + ILVRL_B2_SH(zero, vec10, tmp4, tmp5); + ILVRL_B2_SH(zero, vec11, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + + PCKEV_B3_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, out0, out1, out2); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_W4(out2, 0, 1, 2, 3, dst + 8, dst_stride); dst += (4 * dst_stride); @@ -3011,33 +3399,25 @@ static void hevc_hz_uniwgt_4t_16w_msa(const uint8_t *src, uint32_t loop_cnt; v16u8 out0, out1, out2, out3; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[0]); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; @@ -3046,34 +3426,77 @@ static void hevc_hz_uniwgt_4t_16w_msa(const uint8_t *src, LD_SB4(src + 8, src_stride, src1, src3, src5, src7); src += (4 * src_stride); - XORI_B8_128_SB(src0, src1, src2, src3, src4, src5, src6, src7); - VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec2, vec3); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec4, vec5); VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec6, vec7); - dst4 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, - weight_vec, offset_vec, rnd_vec, - dst4, dst5, dst6, dst7); - - PCKEV_B4_UB(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst12 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + PCKEV_B4_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, tmp7, tmp6, out0, out1, out2, out3); ST_UB4(out0, out1, out2, out3, dst, dst_stride); @@ -3094,31 +3517,24 @@ static void hevc_hz_uniwgt_4t_24w_msa(const uint8_t *src, uint32_t loop_cnt; v16u8 out0, out1, out2; v16i8 src0, src1, src2, src3; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16i8 mask0, mask1, mask2, mask3; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); - weight = weight & 0x0000FFFF; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask0 = LD_SB(&ff_hevc_mask_arr[0]); mask1 = mask0 + 2; @@ -3130,29 +3546,62 @@ static void hevc_hz_uniwgt_4t_24w_msa(const uint8_t *src, LD_SB2(src + 16, src_stride, src1, src3); src += (2 * src_stride); - XORI_B4_128_SB(src0, src1, src2, src3); - VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src0, src1, src0, src1, mask2, mask3, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src2, src3, src2, src3, mask2, mask3, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec0, vec1); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec2, vec3); - dst4 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, - rnd_vec, dst4, dst5); - - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + + PCKEV_B3_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, out0, out1, out2); ST_UB2(out0, out1, dst, dst_stride); ST_D2(out2, 0, 1, dst + 16, dst_stride); dst += (2 * dst_stride); @@ -3172,33 +3621,25 @@ static void hevc_hz_uniwgt_4t_32w_msa(const uint8_t *src, uint32_t loop_cnt; v16u8 out0, out1, out2, out3; v16i8 src0, src1, src2, src3, src4, src5; - v8i16 filt0, filt1; + v8i16 filter_vec, filt0, filt1; v16i8 mask0 = LD_SB(&ff_hevc_mask_arr[0]); v16i8 mask1, mask2, mask3; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= 1; filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); - - weight = weight & 0x0000FFFF; + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); mask1 = mask0 + 2; mask2 = mask0 + 8; @@ -3211,34 +3652,78 @@ static void hevc_hz_uniwgt_4t_32w_msa(const uint8_t *src, LD_SB2(src, 16, src3, src4); src5 = LD_SB(src + 24); src += src_stride; - XORI_B6_128_SB(src0, src1, src2, src3, src4, src5); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src0, src1, src0, src1, mask2, mask3, vec2, vec3); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec4, vec5); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec6, vec7); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); VSHF_B2_SB(src3, src4, src3, src4, mask2, mask3, vec2, vec3); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec4, vec5); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec6, vec7); - dst4 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, - weight_vec, offset_vec, rnd_vec, - dst4, dst5, dst6, dst7); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst12 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + PCKEV_B4_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, tmp7, tmp6, + out0, out1, out2, out3); ST_UB2(out0, out1, dst, 16); dst += dst_stride; ST_UB2(out2, out3, dst, 16); @@ -3261,42 +3746,36 @@ static void hevc_vt_uniwgt_4t_4x2_msa(const uint8_t *src, v16i8 src2110, src4332; v8i16 dst0; v4i32 dst0_r, dst0_l; - v8i16 filt0, filt1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v8i16 tmp0, tmp1, tmp2, tmp3; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB5(src, src_stride, src0, src1, src2, src3, src4); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); ILVR_D2_SB(src21_r, src10_r, src43_r, src32_r, src2110, src4332); - XORI_B2_128_SB(src2110, src4332); - dst0 = HEVC_FILT_4TAP_SH(src2110, src4332, filt0, filt1); - ILVRL_H2_SW(dst0, dst0, dst0_r, dst0_l); - DOTP_SH2_SW(dst0_r, dst0_l, weight_vec, weight_vec, dst0_r, dst0_l); + ILVRL_B2_SH(zero, src2110, tmp0, tmp1); + ILVRL_B2_SH(zero, src4332, tmp2, tmp3); + + dst0_r = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst0_l = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + + MUL2(dst0_r, weight_vec, dst0_l, weight_vec, dst0_r, dst0_l); SRAR_W2_SW(dst0_r, dst0_l, rnd_vec); + ADD2(dst0_r, offset_vec, dst0_l, offset_vec, dst0_r, dst0_l); + CLIP_SW2_0_255(dst0_r, dst0_l); dst0 = __msa_pckev_h((v8i16) dst0_l, (v8i16) dst0_r); - dst0 = __msa_adds_s_h(dst0, offset_vec); - CLIP_SH_0_255(dst0); out = (v16u8) __msa_pckev_b((v16i8) dst0, (v16i8) dst0); ST_W2(out, 0, 1, dst, dst_stride); } @@ -3314,30 +3793,21 @@ static void hevc_vt_uniwgt_4t_4x4_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6; v16i8 src10_r, src32_r, src54_r, src21_r, src43_r, src65_r; v16i8 src2110, src4332, src6554; - v8i16 dst0, dst1; - v8i16 filt0, filt1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5; + v8i16 filter_vec, filt0, filt1; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); @@ -3345,13 +3815,25 @@ static void hevc_vt_uniwgt_4t_4x4_msa(const uint8_t *src, src32_r, src43_r, src54_r, src65_r); ILVR_D3_SB(src21_r, src10_r, src43_r, src32_r, src65_r, src54_r, src2110, src4332, src6554); - XORI_B3_128_SB(src2110, src4332, src6554); - dst0 = HEVC_FILT_4TAP_SH(src2110, src4332, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src4332, src6554, filt0, filt1); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst0, dst1, weight_vec, offset_vec, rnd_vec, - dst0, dst1); - out = (v16u8) __msa_pckev_b((v16i8) dst1, (v16i8) dst0); + ILVRL_B2_SH(zero, src2110, tmp0, tmp1); + ILVRL_B2_SH(zero, src4332, tmp2, tmp3); + ILVRL_B2_SH(zero, src6554, tmp4, tmp5); + + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp4, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp5, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_W4(out, 0, 1, 2, 3, dst, dst_stride); } @@ -3372,35 +3854,26 @@ static void hevc_vt_uniwgt_4t_4x8multiple_msa(const uint8_t *src, v16i8 src21_r, src43_r, src65_r, src87_r, src109_r; v16i8 src2110, src4332, src6554, src8776; v16i8 src10998; - v8i16 dst0, dst1, dst2, dst3, filt0, filt1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v8i16 filter_vec, filt0, filt1; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); src2110 = (v16i8) __msa_ilvr_d((v2i64) src21_r, (v2i64) src10_r); - src2110 = (v16i8) __msa_xori_b((v16u8) src2110, 128); for (loop_cnt = (height >> 3); loop_cnt--;) { LD_SB8(src, src_stride, @@ -3412,17 +3885,36 @@ static void hevc_vt_uniwgt_4t_4x8multiple_msa(const uint8_t *src, ILVR_B2_SB(src9, src8, src10, src9, src98_r, src109_r); ILVR_D4_SB(src43_r, src32_r, src65_r, src54_r, src87_r, src76_r, src109_r, src98_r, src4332, src6554, src8776, src10998); - XORI_B4_128_SB(src4332, src6554, src8776, src10998); - dst0 = HEVC_FILT_4TAP_SH(src2110, src4332, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src4332, src6554, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src6554, src8776, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src8776, src10998, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); + ILVRL_B2_SH(zero, src2110, tmp0, tmp1); + ILVRL_B2_SH(zero, src4332, tmp2, tmp3); + ILVRL_B2_SH(zero, src6554, tmp4, tmp5); + ILVRL_B2_SH(zero, src8776, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp4, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp5, filt0, filt1); + ILVRL_B2_SH(zero, src10998, tmp0, tmp1); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp0, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp1, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); @@ -3468,59 +3960,90 @@ static void hevc_vt_uniwgt_4t_6w_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 src10_r, src32_r, src21_r, src43_r; v16i8 src54_r, src65_r, src76_r, src87_r, src98_r, src109_r; - v8i16 filt0, filt1; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); - XORI_B3_128_SB(src0, src1, src2); - XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); ILVR_B2_SB(src5, src4, src6, src5, src54_r, src65_r); ILVR_B2_SB(src7, src6, src8, src7, src76_r, src87_r); ILVR_B2_SB(src9, src8, src10, src9, src98_r, src109_r); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src54_r, src76_r, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src65_r, src87_r, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(src76_r, src98_r, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(src87_r, src109_r, filt0, filt1); - - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, - weight_vec, offset_vec, rnd_vec, - dst0, dst1, dst2, dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, - weight_vec, offset_vec, rnd_vec, - dst4, dst5, dst6, dst7); - - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src76_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src87_r, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src98_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src109_r, tmp2, tmp3); + dst12 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); + PCKEV_B2_UB(tmp5, tmp4, tmp7, tmp6, out2, out3); ST_W2(out0, 0, 2, dst, dst_stride); ST_H2(out0, 2, 6, dst + 4, dst_stride); ST_W2(out1, 0, 2, dst + 2 * dst_stride, dst_stride); @@ -3544,42 +4067,44 @@ static void hevc_vt_uniwgt_4t_8x2_msa(const uint8_t *src, v16u8 out; v16i8 src0, src1, src2, src3, src4; v16i8 src10_r, src32_r, src21_r, src43_r; - v8i16 dst0, dst1; - v8i16 filt0, filt1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v8i16 filter_vec, filt0, filt1; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB5(src, src_stride, src0, src1, src2, src3, src4); - XORI_B5_128_SB(src0, src1, src2, src3, src4); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst0, dst1, weight_vec, offset_vec, rnd_vec, - dst0, dst1); + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); - out = (v16u8) __msa_pckev_b((v16i8) dst1, (v16i8) dst0); + out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_D2(out, 0, 1, dst, dst_stride); } @@ -3596,45 +4121,56 @@ static void hevc_vt_uniwgt_4t_8x4_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4; v16i8 src10_r, src32_r, src21_r, src43_r; v16i8 src5, src6, src54_r, src65_r; - v8i16 filt0, filt1; - v8i16 dst0, dst1, dst2, dst3; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); - src += (3 * src_stride); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); ILVR_B2_SB(src5, src4, src6, src5, src54_r, src65_r); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); } @@ -3651,52 +4187,74 @@ static void hevc_vt_uniwgt_4t_8x6_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; v16i8 src10_r, src32_r, src54_r, src76_r; v16i8 src21_r, src43_r, src65_r, src87_r; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v8i16 filt0, filt1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v8i16 filter_vec, filt0, filt1; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); LD_SB6(src, src_stride, src3, src4, src5, src6, src7, src8); - XORI_B3_128_SB(src0, src1, src2); - XORI_B6_128_SB(src3, src4, src5, src6, src7, src8); ILVR_B4_SB(src1, src0, src2, src1, src3, src2, src4, src3, src10_r, src21_r, src32_r, src43_r); ILVR_B4_SB(src5, src4, src6, src5, src7, src6, src8, src7, src54_r, src65_r, src76_r, src87_r); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src54_r, src76_r, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src65_r, src87_r, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, dst3); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, rnd_vec, - dst4, dst5); - PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src76_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src87_r, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_B3_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, out0, out1, out2); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_D2(out2, 0, 1, dst + 4 * dst_stride, dst_stride); } @@ -3716,61 +4274,94 @@ static void hevc_vt_uniwgt_4t_8x8mult_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 src10_r, src32_r, src21_r, src43_r; v16i8 src54_r, src65_r, src76_r, src87_r, src98_r, src109_r; - v8i16 filt0, filt1; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); for (loop_cnt = (height >> 3); loop_cnt--;) { LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); src += (8 * src_stride); - XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); ILVR_B2_SB(src5, src4, src6, src5, src54_r, src65_r); ILVR_B2_SB(src7, src6, src8, src7, src76_r, src87_r); ILVR_B2_SB(src9, src8, src10, src9, src98_r, src109_r); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src54_r, src76_r, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src65_r, src87_r, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(src76_r, src98_r, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(src87_r, src109_r, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, - offset_vec, rnd_vec, dst4, dst5, dst6, - dst7); - PCKEV_B2_UB(dst1, dst0, dst3, dst2, out0, out1); - PCKEV_B2_UB(dst5, dst4, dst7, dst6, out2, out3); + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src76_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src87_r, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src98_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src109_r, tmp2, tmp3); + dst12 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); + PCKEV_B2_UB(tmp5, tmp4, tmp7, tmp6, out2, out3); ST_D8(out0, out1, out2, out3, 0, 1, 0, 1, 0, 1, 0, 1, dst, dst_stride); dst += (8 * dst_stride); @@ -3824,34 +4415,25 @@ static void hevc_vt_uniwgt_4t_12w_msa(const uint8_t *src, v16i8 src2110, src4332; v16i8 src54_r, src76_r, src98_r, src65_r, src87_r, src109_r; v16i8 src76_l, src98_l, src87_l, src109_l, src6554, src8776, src10998; - v8i16 filt0, filt1; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8; - v8i16 dst9, dst10, dst11, filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8; + v4i32 dst9, dst10, dst11; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7, tmp8, tmp9; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (1 * src_stride); - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVL_B2_SB(src1, src0, src2, src1, src10_l, src21_l); src2110 = (v16i8) __msa_ilvr_d((v2i64) src21_l, (v2i64) src10_l); @@ -3859,24 +4441,58 @@ static void hevc_vt_uniwgt_4t_12w_msa(const uint8_t *src, for (loop_cnt = 2; loop_cnt--;) { LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); src += (8 * src_stride); - XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); ILVRL_B2_SB(src3, src2, src32_r, src32_l); ILVRL_B2_SB(src4, src3, src43_r, src43_l); ILVRL_B2_SB(src5, src4, src54_r, src54_l); ILVRL_B2_SB(src6, src5, src65_r, src65_l); src4332 = (v16i8) __msa_ilvr_d((v2i64) src43_l, (v2i64) src32_l); src6554 = (v16i8) __msa_ilvr_d((v2i64) src65_l, (v2i64) src54_l); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src2110, src4332, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src4332, src6554, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst4, dst5, weight_vec, offset_vec, - rnd_vec, dst4, dst5); + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src2110, tmp4, tmp5); + ILVRL_B2_SH(zero, src4332, tmp6, tmp7); + ILVRL_B2_SH(zero, src6554, tmp8, tmp9); + dst8 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp6, tmp8, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp7, tmp9, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, dst0, dst1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, dst2, dst3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, dst4, dst5); + PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out0, out1, out2); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_W4(out2, 0, 1, 2, 3, dst + 8, dst_stride); @@ -3888,18 +4504,51 @@ static void hevc_vt_uniwgt_4t_12w_msa(const uint8_t *src, ILVRL_B2_SB(src10, src9, src109_r, src109_l); src8776 = (v16i8) __msa_ilvr_d((v2i64) src87_l, (v2i64) src76_l); src10998 = (v16i8) __msa_ilvr_d((v2i64) src109_l, (v2i64) src98_l); - dst6 = HEVC_FILT_4TAP_SH(src54_r, src76_r, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(src65_r, src87_r, filt0, filt1); - dst8 = HEVC_FILT_4TAP_SH(src76_r, src98_r, filt0, filt1); - dst9 = HEVC_FILT_4TAP_SH(src87_r, src109_r, filt0, filt1); - dst10 = HEVC_FILT_4TAP_SH(src6554, src8776, filt0, filt1); - dst11 = HEVC_FILT_4TAP_SH(src8776, src10998, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst6, dst7, dst8, dst9, weight_vec, - offset_vec, rnd_vec, dst6, dst7, dst8, - dst9); - HEVC_UNIW_RND_CLIP2_MAX_SATU_H(dst10, dst11, weight_vec, offset_vec, - rnd_vec, dst10, dst11); - PCKEV_B3_UB(dst7, dst6, dst9, dst8, dst11, dst10, out3, out4, out5); + + ILVRL_B2_SH(zero, src76_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src87_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src98_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src109_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src8776, tmp4, tmp5); + ILVRL_B2_SH(zero, src10998, tmp6, tmp7); + ILVRL_B2_SH(zero, src6554, tmp8, tmp9); + dst8 = HEVC_FILT_4TAP_SW(tmp8, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp9, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + CLIP_SW4_0_255(dst8, dst9, dst10, dst11); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, dst0, dst1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, dst2, dst3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, dst4, dst5); + + PCKEV_B3_UB(dst1, dst0, dst3, dst2, dst5, dst4, out3, out4, out5); ST_D4(out3, out4, 0, 1, 0, 1, dst, dst_stride); ST_W4(out5, 0, 1, 2, 3, dst + 8, dst_stride); dst += (4 * dst_stride); @@ -3927,60 +4576,94 @@ static void hevc_vt_uniwgt_4t_16w_msa(const uint8_t *src, v16i8 src10_r, src32_r, src21_r, src43_r; v16i8 src10_l, src32_l, src21_l, src43_l; v16i8 src54_r, src54_l, src65_r, src65_l, src6; - v8i16 filt0, filt1; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVL_B2_SB(src1, src0, src2, src1, src10_l, src21_l); for (loop_cnt = (height >> 2); loop_cnt--;) { LD_SB4(src, src_stride, src3, src4, src5, src6); src += (4 * src_stride); - XORI_B4_128_SB(src3, src4, src5, src6); ILVRL_B2_SB(src3, src2, src32_r, src32_l); ILVRL_B2_SB(src4, src3, src43_r, src43_l); ILVRL_B2_SB(src5, src4, src54_r, src54_l); ILVRL_B2_SB(src6, src5, src65_r, src65_l); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src10_l, src32_l, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src21_l, src43_l, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(src32_l, src54_l, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(src43_l, src65_l, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, - offset_vec, rnd_vec, dst4, dst5, dst6, - dst7); - PCKEV_B4_UB(dst4, dst0, dst5, dst1, dst6, dst2, dst7, dst3, out0, out1, + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src10_l, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_l, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_l, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_l, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_l, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_l, tmp2, tmp3); + dst12 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + PCKEV_B4_UB(tmp4, tmp0, tmp5, tmp1, tmp6, tmp2, tmp7, tmp3, out0, out1, out2, out3); ST_UB4(out0, out1, out2, out3, dst, dst_stride); dst += (4 * dst_stride); @@ -4010,36 +4693,27 @@ static void hevc_vt_uniwgt_4t_24w_msa(const uint8_t *src, v16i8 src10_r, src32_r, src54_r, src21_r, src43_r, src65_r; v16i8 src10_l, src32_l, src54_l, src21_l, src43_l, src65_l; v16i8 src87_r, src98_r, src109_r, src1110_r, src1211_r, src1312_r; - v8i16 filt0, filt1; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8, dst9, dst10; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec, dst11; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v4i32 dst16, dst17, dst18, dst19, dst20, dst21, dst22, dst23; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); LD_SB3(src + 16, src_stride, src7, src8, src9); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); - XORI_B3_128_SB(src7, src8, src9); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVL_B2_SB(src1, src0, src2, src1, src10_l, src21_l); ILVR_B2_SB(src8, src7, src9, src8, src87_r, src98_r); @@ -4048,38 +4722,103 @@ static void hevc_vt_uniwgt_4t_24w_msa(const uint8_t *src, LD_SB4(src, src_stride, src3, src4, src5, src6); LD_SB4(src + 16, src_stride, src10, src11, src12, src13); src += (4 * src_stride); - XORI_B4_128_SB(src3, src4, src5, src6); - XORI_B4_128_SB(src10, src11, src12, src13); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); ILVL_B2_SB(src3, src2, src4, src3, src32_l, src43_l); ILVRL_B2_SB(src5, src4, src54_r, src54_l); ILVRL_B2_SB(src6, src5, src65_r, src65_l); ILVR_B2_SB(src10, src9, src11, src10, src109_r, src1110_r); ILVR_B2_SB(src12, src11, src13, src12, src1211_r, src1312_r); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src32_r, src54_r, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src43_r, src65_r, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src10_l, src32_l, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src21_l, src43_l, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(src32_l, src54_l, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(src43_l, src65_l, filt0, filt1); - dst8 = HEVC_FILT_4TAP_SH(src87_r, src109_r, filt0, filt1); - dst9 = HEVC_FILT_4TAP_SH(src98_r, src1110_r, filt0, filt1); - dst10 = HEVC_FILT_4TAP_SH(src109_r, src1211_r, filt0, filt1); - dst11 = HEVC_FILT_4TAP_SH(src1110_r, src1312_r, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, - offset_vec, rnd_vec, dst4, dst5, dst6, - dst7); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst8, dst9, dst10, dst11, weight_vec, - offset_vec, rnd_vec, dst8, dst9, dst10, - dst11); - PCKEV_B4_UB(dst4, dst0, dst5, dst1, dst6, dst2, dst7, dst3, out0, out1, + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_r, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src10_l, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_l, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_l, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_l, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src54_l, tmp0, tmp1); + ILVRL_B2_SH(zero, src65_l, tmp2, tmp3); + dst12 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + ILVRL_B2_SH(zero, src87_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src98_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src109_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src1110_r, tmp6, tmp7); + dst16 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst17 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst18 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst19 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src1211_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src1312_r, tmp2, tmp3); + dst20 = HEVC_FILT_4TAP_SW(tmp4, tmp0, filt0, filt1); + dst21 = HEVC_FILT_4TAP_SW(tmp5, tmp1, filt0, filt1); + dst22 = HEVC_FILT_4TAP_SW(tmp6, tmp2, filt0, filt1); + dst23 = HEVC_FILT_4TAP_SW(tmp7, tmp3, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + MUL4(dst16, weight_vec, dst17, weight_vec, dst18, weight_vec, dst19, + weight_vec, dst16, dst17, dst18, dst19); + MUL4(dst20, weight_vec, dst21, weight_vec, dst22, weight_vec, + dst23, weight_vec, dst20, dst21, dst22, dst23); + SRAR_W4_SW(dst16, dst17, dst18, dst19, rnd_vec); + SRAR_W4_SW(dst20, dst21, dst22, dst23, rnd_vec); + ADD4(dst16, offset_vec, dst17, offset_vec, dst18, offset_vec, dst19, + offset_vec, dst16, dst17, dst18, dst19); + ADD4(dst20, offset_vec, dst21, offset_vec, dst22, offset_vec, + dst23, offset_vec, dst20, dst21, dst22, dst23); + CLIP_SW8_0_255(dst16, dst17, dst18, dst19, dst20, dst21, dst22, dst23); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + PCKEV_B4_UB(tmp4, tmp0, tmp5, tmp1, tmp6, tmp2, tmp7, tmp3, out0, out1, out2, out3); - PCKEV_B2_UB(dst9, dst8, dst11, dst10, out4, out5); + + PCKEV_H2_SH(dst17, dst16, dst19, dst18, tmp0, tmp1); + PCKEV_H2_SH(dst21, dst20, dst23, dst22, tmp2, tmp3); + + PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out4, out5); ST_UB4(out0, out1, out2, out3, dst, dst_stride); ST_D4(out4, out5, 0, 1, 0, 1, dst + 16, dst_stride); dst += (4 * dst_stride); @@ -4110,37 +4849,28 @@ static void hevc_vt_uniwgt_4t_32w_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9; v16i8 src10_r, src32_r, src76_r, src98_r; v16i8 src21_r, src43_r, src65_r, src87_r; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; v16i8 src10_l, src32_l, src76_l, src98_l; v16i8 src21_l, src43_l, src65_l, src87_l; - v8i16 filt0, filt1; - v8i16 filter_vec, weight_vec_h, offset_vec, denom_vec; - v4i32 weight_vec, rnd_vec; + v8i16 filter_vec, filt0, filt1; + v4i32 weight_vec, rnd_vec, offset_vec; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v4i32 dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v8i16 zero = { 0 }; src -= src_stride; - weight = weight & 0x0000FFFF; - weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - weight *= 128; - rnd_val -= 6; - - weight_vec_h = __msa_fill_h(weight); - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val); - - weight_vec_h = __msa_srar_h(weight_vec_h, denom_vec); - offset_vec = __msa_adds_s_h(offset_vec, weight_vec_h); + offset_vec = __msa_fill_w(offset); filter_vec = LD_SH(filter); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); LD_SB3(src, src_stride, src0, src1, src2); LD_SB3(src + 16, src_stride, src5, src6, src7); src += (3 * src_stride); - XORI_B6_128_SB(src0, src1, src2, src5, src6, src7); ILVR_B2_SB(src1, src0, src2, src1, src10_r, src21_r); ILVL_B2_SB(src1, src0, src2, src1, src10_l, src21_l); ILVR_B2_SB(src6, src5, src7, src6, src65_r, src76_r); @@ -4150,26 +4880,75 @@ static void hevc_vt_uniwgt_4t_32w_msa(const uint8_t *src, LD_SB2(src, src_stride, src3, src4); LD_SB2(src + 16, src_stride, src8, src9); src += (2 * src_stride); - XORI_B4_128_SB(src3, src4, src8, src9); ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); ILVL_B2_SB(src3, src2, src4, src3, src32_l, src43_l); ILVRL_B2_SB(src8, src7, src87_r, src87_l); ILVRL_B2_SB(src9, src8, src98_r, src98_l); - dst0 = HEVC_FILT_4TAP_SH(src10_r, src32_r, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(src21_r, src43_r, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(src10_l, src32_l, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(src21_l, src43_l, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(src65_r, src87_r, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(src76_r, src98_r, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(src65_l, src87_l, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(src76_l, src98_l, filt0, filt1); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst0, dst1, dst2, dst3, weight_vec, - offset_vec, rnd_vec, dst0, dst1, dst2, - dst3); - HEVC_UNIW_RND_CLIP4_MAX_SATU_H(dst4, dst5, dst6, dst7, weight_vec, - offset_vec, rnd_vec, dst4, dst5, dst6, - dst7); - PCKEV_B4_UB(dst2, dst0, dst3, dst1, dst6, dst4, dst7, dst5, out0, out1, + + ILVRL_B2_SH(zero, src10_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_r, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + + ILVRL_B2_SH(zero, src10_l, tmp0, tmp1); + ILVRL_B2_SH(zero, src21_l, tmp2, tmp3); + ILVRL_B2_SH(zero, src32_l, tmp4, tmp5); + ILVRL_B2_SH(zero, src43_l, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + + ILVRL_B2_SH(zero, src65_r, tmp0, tmp1); + ILVRL_B2_SH(zero, src76_r, tmp2, tmp3); + ILVRL_B2_SH(zero, src87_r, tmp4, tmp5); + ILVRL_B2_SH(zero, src98_r, tmp6, tmp7); + dst8 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst11 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, src65_l, tmp0, tmp1); + ILVRL_B2_SH(zero, src76_l, tmp2, tmp3); + ILVRL_B2_SH(zero, src87_l, tmp4, tmp5); + ILVRL_B2_SH(zero, src98_l, tmp6, tmp7); + dst12 = HEVC_FILT_4TAP_SW(tmp0, tmp4, filt0, filt1); + dst13 = HEVC_FILT_4TAP_SW(tmp1, tmp5, filt0, filt1); + dst14 = HEVC_FILT_4TAP_SW(tmp2, tmp6, filt0, filt1); + dst15 = HEVC_FILT_4TAP_SW(tmp3, tmp7, filt0, filt1); + + MUL4(dst0, weight_vec, dst1, weight_vec, dst2, weight_vec, dst3, + weight_vec, dst0, dst1, dst2, dst3); + MUL4(dst4, weight_vec, dst5, weight_vec, dst6, weight_vec, + dst7, weight_vec, dst4, dst5, dst6, dst7); + SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, + dst7, offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); + + MUL4(dst8, weight_vec, dst9, weight_vec, dst10, weight_vec, dst11, + weight_vec, dst8, dst9, dst10, dst11); + MUL4(dst12, weight_vec, dst13, weight_vec, dst14, weight_vec, + dst15, weight_vec, dst12, dst13, dst14, dst15); + SRAR_W4_SW(dst8, dst9, dst10, dst11, rnd_vec); + SRAR_W4_SW(dst12, dst13, dst14, dst15, rnd_vec); + ADD4(dst8, offset_vec, dst9, offset_vec, dst10, offset_vec, dst11, + offset_vec, dst8, dst9, dst10, dst11); + ADD4(dst12, offset_vec, dst13, offset_vec, dst14, offset_vec, + dst15, offset_vec, dst12, dst13, dst14, dst15); + CLIP_SW8_0_255(dst8, dst9, dst10, dst11, dst12, dst13, dst14, dst15); + + PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); + PCKEV_H2_SH(dst5, dst4, dst7, dst6, tmp2, tmp3); + PCKEV_H2_SH(dst9, dst8, dst11, dst10, tmp4, tmp5); + PCKEV_H2_SH(dst13, dst12, dst15, dst14, tmp6, tmp7); + PCKEV_B4_UB(tmp2, tmp0, tmp3, tmp1, tmp6, tmp4, tmp7, tmp5, out0, out1, out2, out3); ST_UB2(out0, out2, dst, 16); dst += dst_stride; @@ -4206,49 +4985,54 @@ static void hevc_hv_uniwgt_4t_4x2_msa(const uint8_t *src, v16i8 mask1; v8i16 filt_h0, filt_h1, filter_vec, tmp; v16i8 vec0, vec1, vec2, vec3, vec4, vec5; - v8i16 dst20, dst31, dst42, dst10, dst32, dst21, dst43; - v8i16 offset_vec, const_128, denom_vec; - v4i32 dst0, dst1, weight_vec, rnd_vec; + v8i16 dst10, dst21, dst32, dst43; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); LD_SB5(src, src_stride, src0, src1, src2, src3, src4); - XORI_B5_128_SB(src0, src1, src2, src3, src4); VSHF_B2_SB(src0, src2, src0, src2, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src3, src1, src3, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src4, src2, src4, mask0, mask1, vec4, vec5); - dst20 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst31 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst42 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - ILVRL_H2_SH(dst31, dst20, dst10, dst32); - ILVRL_H2_SH(dst42, dst31, dst21, dst43); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVL_B2_SH(zero, vec4, zero, vec5, tmp1, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + + ILVEV_H2_SH(dst0, dst1, dst2, dst3, dst10, dst32); + ILVEV_H2_SH(dst1, dst2, dst3, dst4, dst21, dst43); dst0 = HEVC_FILT_4TAP(dst10, dst32, filt_h0, filt_h1); dst1 = HEVC_FILT_4TAP(dst21, dst43, filt_h0, filt_h1); dst0 >>= 6; dst1 >>= 6; MUL2(dst0, weight_vec, dst1, weight_vec, dst0, dst1); SRAR_W2_SW(dst0, dst1, rnd_vec); + ADD2(dst0, offset_vec, dst1, offset_vec, dst0, dst1); + CLIP_SW2_0_255(dst0, dst1); tmp = __msa_pckev_h((v8i16) dst1, (v8i16) dst0); - tmp += offset_vec; - CLIP_SH_0_255(tmp); out = (v16u8) __msa_pckev_b((v16i8) tmp, (v16i8) tmp); ST_W2(out, 0, 1, dst, dst_stride); } @@ -4266,47 +5050,55 @@ static void hevc_hv_uniwgt_4t_4x4_msa(const uint8_t *src, v16u8 out; v16i8 src0, src1, src2, src3, src4, src5, src6; v8i16 filt0, filt1; - v8i16 filt_h0, filt_h1, filter_vec, tmp0, tmp1; + v8i16 filt_h0, filt_h1, filter_vec; v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst30, dst41, dst52, dst63, dst10, dst32, dst54, dst21, dst43, dst65; - v8i16 offset_vec, const_128, denom_vec; - v4i32 dst0, dst1, dst2, dst3, weight_vec, rnd_vec; + v8i16 dst10, dst32, dst54, dst21, dst43, dst65; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); VSHF_B2_SB(src0, src3, src0, src3, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src4, src1, src4, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src5, src2, src5, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src6, src3, src6, mask0, mask1, vec6, vec7); - dst30 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst41 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst52 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst63 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - ILVRL_H2_SH(dst41, dst30, dst10, dst43); - ILVRL_H2_SH(dst52, dst41, dst21, dst54); - ILVRL_H2_SH(dst63, dst52, dst32, dst65); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst4 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVL_B2_SH(zero, vec6, zero, vec7, tmp5, tmp7); + dst2 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst0, dst1, dst3, dst4, dst10, dst43); + ILVEV_H2_SH(dst1, dst2, dst4, dst5, dst21, dst54); + ILVEV_H2_SH(dst2, dst3, dst5, dst6, dst32, dst65); dst0 = HEVC_FILT_4TAP(dst10, dst32, filt_h0, filt_h1); dst1 = HEVC_FILT_4TAP(dst21, dst43, filt_h0, filt_h1); dst2 = HEVC_FILT_4TAP(dst32, dst54, filt_h0, filt_h1); @@ -4315,9 +5107,10 @@ static void hevc_hv_uniwgt_4t_4x4_msa(const uint8_t *src, MUL2(dst0, weight_vec, dst1, weight_vec, dst0, dst1); MUL2(dst2, weight_vec, dst3, weight_vec, dst2, dst3); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + CLIP_SW4_0_255(dst0, dst1, dst2, dst3); PCKEV_H2_SH(dst1, dst0, dst3, dst2, tmp0, tmp1); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - CLIP_SH2_0_255(tmp0, tmp1); out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_W4(out, 0, 1, 2, 3, dst, dst_stride); } @@ -4339,65 +5132,78 @@ static void hevc_hv_uniwgt_4t_4multx8mult_msa(const uint8_t *src, v8i16 filt0, filt1; v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); v16i8 mask1; - v8i16 filt_h0, filt_h1, filter_vec, tmp0, tmp1, tmp2, tmp3; + v8i16 filter_vec, filt_h0, filt_h1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst10, dst21, dst22, dst73, dst84, dst95, dst106; v8i16 dst10_r, dst32_r, dst54_r, dst76_r; v8i16 dst21_r, dst43_r, dst65_r, dst87_r; - v8i16 dst98_r, dst109_r, offset_vec, const_128, denom_vec; - v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, weight_vec, rnd_vec; + v8i16 dst98_r, dst109_r; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8, dst9, dst10; + v4i32 offset_vec, weight_vec, rnd_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); VSHF_B2_SB(src0, src1, src0, src1, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src2, src1, src2, mask0, mask1, vec2, vec3); - dst10 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst21 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - ILVRL_H2_SH(dst21, dst10, dst10_r, dst21_r); - dst22 = (v8i16) __msa_splati_d((v2i64) dst21, 1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVL_B2_SH(zero, vec2, zero, vec3, tmp5, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + ILVEV_H2_SH(dst0, dst1, dst1, dst2, dst10_r, dst21_r); for (loop_cnt = height >> 3; loop_cnt--;) { LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); src += (8 * src_stride); - XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); VSHF_B2_SB(src3, src7, src3, src7, mask0, mask1, vec0, vec1); VSHF_B2_SB(src4, src8, src4, src8, mask0, mask1, vec2, vec3); VSHF_B2_SB(src5, src9, src5, src9, mask0, mask1, vec4, vec5); VSHF_B2_SB(src6, src10, src6, src10, mask0, mask1, vec6, vec7); - dst73 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst84 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst95 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst106 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - dst32_r = __msa_ilvr_h(dst73, dst22); - ILVRL_H2_SH(dst84, dst73, dst43_r, dst87_r); - ILVRL_H2_SH(dst95, dst84, dst54_r, dst98_r); - ILVRL_H2_SH(dst106, dst95, dst65_r, dst109_r); - dst22 = (v8i16) __msa_splati_d((v2i64) dst73, 1); - dst76_r = __msa_ilvr_h(dst22, dst106); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst3 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst8 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst5 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + dst32_r = __msa_ilvev_h(dst3, dst2); + ILVEV_H2_SH(dst3, dst4, dst7, dst8, dst43_r, dst87_r); + ILVEV_H2_SH(dst4, dst5, dst8, dst9, dst54_r, dst98_r); + ILVEV_H2_SH(dst5, dst6, dst9, dst10, dst65_r, dst109_r); + dst76_r = __msa_ilvev_h(dst7, dst6); dst0 = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst1 = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); dst2 = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); @@ -4414,18 +5220,20 @@ static void hevc_hv_uniwgt_4t_4multx8mult_msa(const uint8_t *src, MUL2(dst6, weight_vec, dst7, weight_vec, dst6, dst7); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD4(dst0, offset_vec, dst1, offset_vec, dst2, offset_vec, dst3, + offset_vec, dst0, dst1, dst2, dst3); + ADD4(dst4, offset_vec, dst5, offset_vec, dst6, offset_vec, dst7, + offset_vec, dst4, dst5, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); dst10_r = dst98_r; dst21_r = dst109_r; - dst22 = (v8i16) __msa_splati_d((v2i64) dst106, 1); + dst2 = dst10; } } @@ -4473,75 +5281,103 @@ static void hevc_hv_uniwgt_4t_6w_msa(const uint8_t *src, v16i8 mask1; v8i16 filt_h0, filt_h1, filter_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dsth0, dsth1, dsth2, dsth3, dsth4, dsth5, dsth6, dsth7, dsth8, dsth9; - v8i16 dsth10, tmp0, tmp1, tmp2, tmp3, tmp4, tmp5; + v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8, dst9; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst98_r, dst21_r, dst43_r; v8i16 dst65_r, dst87_r, dst109_r, dst10_l, dst32_l, dst54_l, dst76_l; v8i16 dst98_l, dst21_l, dst43_l, dst65_l, dst87_l, dst109_l; v8i16 dst1021_l, dst3243_l, dst5465_l, dst7687_l, dst98109_l; - v8i16 offset_vec, const_128, denom_vec; v4i32 dst0_r, dst1_r, dst2_r, dst3_r, dst4_r, dst5_r, dst6_r, dst7_r; - v4i32 dst0_l, dst1_l, dst2_l, dst3_l, weight_vec, rnd_vec; + v4i32 dst0_l, dst1_l, dst2_l, dst3_l, weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - dsth0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dsth1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dsth2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - ILVRL_H2_SH(dsth1, dsth0, dst10_r, dst10_l); - ILVRL_H2_SH(dsth2, dsth1, dst21_r, dst21_l); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst21_r, dst21_l); LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); - XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); - dsth3 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dsth4 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dsth5 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dsth6 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst6 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst8 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst32_r, dst32_l); + ILVEV_H2_SH(dst6, dst8, dst7, dst9, dst43_r, dst43_l); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst8, dst0, dst9, dst1, dst54_r, dst54_l); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst65_r, dst65_l); VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec0, vec1); VSHF_B2_SB(src8, src8, src8, src8, mask0, mask1, vec2, vec3); VSHF_B2_SB(src9, src9, src9, src9, mask0, mask1, vec4, vec5); VSHF_B2_SB(src10, src10, src10, src10, mask0, mask1, vec6, vec7); - dsth7 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dsth8 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dsth9 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dsth10 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - ILVRL_H2_SH(dsth3, dsth2, dst32_r, dst32_l); - ILVRL_H2_SH(dsth4, dsth3, dst43_r, dst43_l); - ILVRL_H2_SH(dsth5, dsth4, dst54_r, dst54_l); - ILVRL_H2_SH(dsth6, dsth5, dst65_r, dst65_l); - ILVRL_H2_SH(dsth7, dsth6, dst76_r, dst76_l); - ILVRL_H2_SH(dsth8, dsth7, dst87_r, dst87_l); - ILVRL_H2_SH(dsth9, dsth8, dst98_r, dst98_l); - ILVRL_H2_SH(dsth10, dsth9, dst109_r, dst109_l); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst76_r, dst76_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst87_r, dst87_l); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst6, dst0, dst7, dst1, dst98_r, dst98_l); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst109_r, dst109_l); PCKEV_D2_SH(dst21_l, dst10_l, dst43_l, dst32_l, dst1021_l, dst3243_l); PCKEV_D2_SH(dst65_l, dst54_l, dst87_l, dst76_l, dst5465_l, dst7687_l); dst98109_l = (v8i16) __msa_pckev_d((v2i64) dst109_l, (v2i64) dst98_l); @@ -4569,14 +5405,18 @@ static void hevc_hv_uniwgt_4t_6w_msa(const uint8_t *src, SRAR_W4_SW(dst0_r, dst1_r, dst2_r, dst3_r, rnd_vec); SRAR_W4_SW(dst4_r, dst5_r, dst6_r, dst7_r, rnd_vec); SRAR_W4_SW(dst0_l, dst1_l, dst2_l, dst3_l, rnd_vec); + ADD4(dst0_r, offset_vec, dst1_r, offset_vec, dst2_r, offset_vec, dst3_r, + offset_vec, dst0_r, dst1_r, dst2_r, dst3_r); + ADD4(dst4_r, offset_vec, dst5_r, offset_vec, dst6_r, offset_vec, dst7_r, + offset_vec, dst4_r, dst5_r, dst6_r, dst7_r); + ADD4(dst0_l, offset_vec, dst1_l, offset_vec, dst2_l, offset_vec, dst3_l, + offset_vec, dst0_l, dst1_l, dst2_l, dst3_l); + CLIP_SW8_0_255(dst1_r, dst0_r, dst3_r, dst2_r, + dst4_r, dst5_r, dst6_r, dst7_r); + CLIP_SW4_0_255(dst0_l, dst1_l, dst2_l, dst3_l); PCKEV_H2_SH(dst1_r, dst0_r, dst3_r, dst2_r, tmp0, tmp1); PCKEV_H2_SH(dst5_r, dst4_r, dst7_r, dst6_r, tmp2, tmp3); PCKEV_H2_SH(dst1_l, dst0_l, dst3_l, dst2_l, tmp4, tmp5); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - ADD2(tmp4, offset_vec, tmp5, offset_vec, tmp4, tmp5); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); - CLIP_SH2_0_255(tmp4, tmp5); PCKEV_B3_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, out0, out1, out2); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); ST_H8(out2, 0, 1, 2, 3, 4, 5, 6, 7, dst + 4, dst_stride); @@ -4599,50 +5439,63 @@ static void hevc_hv_uniwgt_4t_8x2_msa(const uint8_t *src, v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9; - v8i16 dst0, dst1, dst2, dst3, dst4; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; v4i32 dst0_r, dst0_l, dst1_r, dst1_l; v8i16 dst10_r, dst32_r, dst21_r, dst43_r; v8i16 dst10_l, dst32_l, dst21_l, dst43_l; - v8i16 tmp0, tmp1; - v8i16 offset_vec, const_128, denom_vec; - v4i32 weight_vec, rnd_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); LD_SB5(src, src_stride, src0, src1, src2, src3, src4); - XORI_B5_128_SB(src0, src1, src2, src3, src4); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec8, vec9); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(vec8, vec9, filt0, filt1); - ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); - ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst21_r, dst21_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst32_r, dst32_l); + ILVRL_B2_SH(zero, vec8, tmp0, tmp1); + ILVRL_B2_SH(zero, vec9, tmp2, tmp3); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + ILVEV_H2_SH(dst6, dst0, dst7, dst1, dst43_r, dst43_l); + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); @@ -4651,9 +5504,10 @@ static void hevc_hv_uniwgt_4t_8x2_msa(const uint8_t *src, MUL2(dst0_r, weight_vec, dst1_r, weight_vec, dst0_r, dst1_r); MUL2(dst0_l, weight_vec, dst1_l, weight_vec, dst0_l, dst1_l); SRAR_W4_SW(dst0_r, dst0_l, dst1_r, dst1_l, rnd_vec); + ADD4(dst0_r, offset_vec, dst0_l, offset_vec, dst1_r, offset_vec, + dst1_l, offset_vec, dst0_r, dst0_l, dst1_r, dst1_l); + CLIP_SW4_0_255(dst0_r, dst0_l, dst1_r, dst1_l); PCKEV_H2_SH(dst0_l, dst0_r, dst1_l, dst1_r, tmp0, tmp1); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - CLIP_SH2_0_255(tmp0, tmp1); out = (v16u8) __msa_pckev_b((v16i8) tmp1, (v16i8) tmp0); ST_D2(out, 0, 1, dst, dst_stride); } @@ -4674,21 +5528,22 @@ static void hevc_hv_uniwgt_4t_8multx4_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6, mask0, mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v8i16 filt0, filt1, filt_h0, filt_h1, filter_vec; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, tmp0, tmp1, tmp2, tmp3; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; v8i16 dst10_r, dst32_r, dst54_r, dst21_r, dst43_r, dst65_r; v8i16 dst10_l, dst32_l, dst54_l, dst21_l, dst43_l, dst65_l; - v8i16 offset_vec, const_128, denom_vec; v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; - v4i32 weight_vec, rnd_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask0 = LD_SB(ff_hevc_mask_arr); @@ -4696,36 +5551,53 @@ static void hevc_hv_uniwgt_4t_8multx4_msa(const uint8_t *src, weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); for (cnt = width8mult; cnt--;) { LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); src += 8; - XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); - ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst21_r, dst21_l); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); - dst3 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); - ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); - ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst4, dst0, dst5, dst1, dst32_r, dst32_l); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst43_r, dst43_l); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst54_r, dst54_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst65_r, dst65_l); + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); @@ -4742,11 +5614,14 @@ static void hevc_hv_uniwgt_4t_8multx4_msa(const uint8_t *src, MUL2(dst2_l, weight_vec, dst3_l, weight_vec, dst2_l, dst3_l); SRAR_W4_SW(dst0_r, dst0_l, dst1_r, dst1_l, rnd_vec); SRAR_W4_SW(dst2_r, dst2_l, dst3_r, dst3_l, rnd_vec); + ADD2(dst0_r, offset_vec, dst1_r, offset_vec, dst0_r, dst1_r); + ADD2(dst2_r, offset_vec, dst3_r, offset_vec, dst2_r, dst3_r); + ADD2(dst0_l, offset_vec, dst1_l, offset_vec, dst0_l, dst1_l); + ADD2(dst2_l, offset_vec, dst3_l, offset_vec, dst2_l, dst3_l); + CLIP_SW8_0_255(dst0_r, dst0_l, dst1_r, dst1_l, + dst2_r, dst2_l, dst3_r, dst3_l); PCKEV_H4_SH(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, dst3_r, tmp0, tmp1, tmp2, tmp3); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); dst += 8; @@ -4771,41 +5646,35 @@ static void hevc_hv_uniwgt_4t_8x6_msa(const uint8_t *src, v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9; v16i8 vec10, vec11, vec12, vec13, vec14, vec15, vec16, vec17; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; - v4i32 dst4_r, dst4_l, dst5_r, dst5_l, weight_vec, rnd_vec; + v4i32 dst4_r, dst4_l, dst5_r, dst5_l, weight_vec, rnd_vec, offset_vec; v8i16 dst10_r, dst32_r, dst10_l, dst32_l; v8i16 dst21_r, dst43_r, dst21_l, dst43_l; v8i16 dst54_r, dst54_l, dst65_r, dst65_l; v8i16 dst76_r, dst76_l, dst87_r, dst87_l; - v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5; - v8i16 offset_vec, const_128, denom_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); LD_SB5(src, src_stride, src0, src1, src2, src3, src4); src += (5 * src_stride); LD_SB4(src, src_stride, src5, src6, src7, src8); - XORI_B5_128_SB(src0, src1, src2, src3, src4); - XORI_B4_128_SB(src5, src6, src7, src8); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); @@ -4815,23 +5684,51 @@ static void hevc_hv_uniwgt_4t_8x6_msa(const uint8_t *src, VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec12, vec13); VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec14, vec15); VSHF_B2_SB(src8, src8, src8, src8, mask0, mask1, vec16, vec17); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst3 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(vec8, vec9, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec10, vec11, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec12, vec13, filt0, filt1); - dst7 = HEVC_FILT_4TAP_SH(vec14, vec15, filt0, filt1); - dst8 = HEVC_FILT_4TAP_SH(vec16, vec17, filt0, filt1); - ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); - ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); - ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); - ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); - ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); - ILVRL_H2_SH(dst8, dst7, dst87_r, dst87_l); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst21_r, dst21_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst32_r, dst32_l); + ILVRL_B2_SH(zero, vec8, tmp0, tmp1); + ILVRL_B2_SH(zero, vec9, tmp2, tmp3); + ILVRL_B2_SH(zero, vec10, tmp4, tmp5); + ILVRL_B2_SH(zero, vec11, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst6, dst0, dst7, dst1, dst43_r, dst43_l); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst54_r, dst54_l); + ILVRL_B2_SH(zero, vec12, tmp0, tmp1); + ILVRL_B2_SH(zero, vec13, tmp2, tmp3); + ILVRL_B2_SH(zero, vec14, tmp4, tmp5); + ILVRL_B2_SH(zero, vec15, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst65_r, dst65_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst76_r, dst76_l); + ILVRL_B2_SH(zero, vec16, tmp0, tmp1); + ILVRL_B2_SH(zero, vec17, tmp2, tmp3); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + ILVEV_H2_SH(dst6, dst0, dst7, dst1, dst87_r, dst87_l); dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); @@ -4856,14 +5753,18 @@ static void hevc_hv_uniwgt_4t_8x6_msa(const uint8_t *src, SRAR_W4_SW(dst0_r, dst0_l, dst1_r, dst1_l, rnd_vec); SRAR_W4_SW(dst2_r, dst2_l, dst3_r, dst3_l, rnd_vec); SRAR_W4_SW(dst4_r, dst4_l, dst5_r, dst5_l, rnd_vec); + ADD2(dst0_r, offset_vec, dst1_r, offset_vec, dst0_r, dst1_r); + ADD2(dst2_r, offset_vec, dst3_r, offset_vec, dst2_r, dst3_r); + ADD2(dst4_r, offset_vec, dst5_r, offset_vec, dst4_r, dst5_r); + ADD2(dst0_l, offset_vec, dst1_l, offset_vec, dst0_l, dst1_l); + ADD2(dst2_l, offset_vec, dst3_l, offset_vec, dst2_l, dst3_l); + ADD2(dst4_l, offset_vec, dst5_l, offset_vec, dst4_l, dst5_l); + CLIP_SW8_0_255(dst0_r, dst1_r, dst2_r, dst3_r, + dst4_r, dst5_r, dst0_l, dst1_l); + CLIP_SW4_0_255(dst2_l, dst3_l, dst4_l, dst5_l); PCKEV_H4_SH(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, dst3_r, tmp0, tmp1, tmp2, tmp3); PCKEV_H2_SH(dst4_l, dst4_r, dst5_l, dst5_r, tmp4, tmp5); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - ADD2(tmp4, offset_vec, tmp5, offset_vec, tmp4, tmp5); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); - CLIP_SH2_0_255(tmp4, tmp5); PCKEV_B3_UB(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, out0, out1, out2); ST_D4(out0, out1, 0, 1, 0, 1, dst, dst_stride); ST_D2(out2, 0, 1, dst + 4 * dst_stride, dst_stride); @@ -4891,33 +5792,30 @@ static void hevc_hv_uniwgt_4t_8multx4mult_msa(const uint8_t *src, v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, tmp0, tmp1, tmp2, tmp3; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; v8i16 dst10_r, dst32_r, dst54_r, dst21_r, dst43_r, dst65_r; v8i16 dst10_l, dst32_l, dst54_l, dst21_l, dst43_l, dst65_l; v4i32 dst0_r, dst0_l, dst1_r, dst1_l; - v8i16 offset_vec, const_128, denom_vec; v4i32 dst2_r, dst2_l, dst3_r, dst3_l; - v4i32 weight_vec, rnd_vec; + v4i32 weight_vec, rnd_vec, offset_vec; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); for (cnt = width8mult; cnt--;) { src_tmp = src; @@ -4925,35 +5823,55 @@ static void hevc_hv_uniwgt_4t_8multx4mult_msa(const uint8_t *src, LD_SB3(src_tmp, src_stride, src0, src1, src2); src_tmp += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - dst0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); - ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst10_r, dst10_l); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst21_r, dst21_l); for (loop_cnt = height >> 2; loop_cnt--;) { LD_SB4(src_tmp, src_stride, src3, src4, src5, src6); src_tmp += (4 * src_stride); - XORI_B4_128_SB(src3, src4, src5, src6); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); - dst3 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst4 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst5 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst6 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); - ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); - ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst4, dst0, dst5, dst1, dst32_r, dst32_l); + ILVEV_H2_SH(dst0, dst2, dst1, dst3, dst43_r, dst43_l); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dst2, dst4, dst3, dst5, dst54_r, dst54_l); + ILVEV_H2_SH(dst4, dst6, dst5, dst7, dst65_r, dst65_l); dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); @@ -4970,11 +5888,14 @@ static void hevc_hv_uniwgt_4t_8multx4mult_msa(const uint8_t *src, MUL2(dst2_l, weight_vec, dst3_l, weight_vec, dst2_l, dst3_l); SRAR_W4_SW(dst0_r, dst0_l, dst1_r, dst1_l, rnd_vec); SRAR_W4_SW(dst2_r, dst2_l, dst3_r, dst3_l, rnd_vec); + ADD2(dst0_r, offset_vec, dst1_r, offset_vec, dst0_r, dst1_r); + ADD2(dst2_r, offset_vec, dst3_r, offset_vec, dst2_r, dst3_r); + ADD2(dst0_l, offset_vec, dst1_l, offset_vec, dst0_l, dst1_l); + ADD2(dst2_l, offset_vec, dst3_l, offset_vec, dst2_l, dst3_l); + CLIP_SW8_0_255(dst0_r, dst0_l, dst1_r, dst1_l, + dst2_r, dst2_l, dst3_r, dst3_l); PCKEV_H4_SH(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, dst3_r, tmp0, tmp1, tmp2, tmp3); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst_tmp, dst_stride); dst_tmp += (4 * dst_stride); @@ -4983,7 +5904,8 @@ static void hevc_hv_uniwgt_4t_8multx4mult_msa(const uint8_t *src, dst10_l = dst54_l; dst21_r = dst65_r; dst21_l = dst65_l; - dst2 = dst6; + dst4 = dst6; + dst5 = dst7; } src += 8; @@ -5040,24 +5962,25 @@ static void hevc_hv_uniwgt_4t_12w_msa(const uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 mask0, mask1, mask2, mask3; - v8i16 filt0, filt1, filt_h0, filt_h1, filter_vec, tmp0, tmp1, tmp2, tmp3; - v8i16 dsth0, dsth1, dsth2, dsth3, dsth4, dsth5, dsth6; - v8i16 dst10, dst21, dst22, dst73, dst84, dst95, dst106; + v8i16 filt0, filt1, filt_h0, filt_h1, filter_vec; + v4i32 dsth0, dsth1, dsth2, dsth3, dsth4, dsth5, dsth6, dsth7; v8i16 dst76_r, dst98_r, dst87_r, dst109_r; v8i16 dst10_r, dst32_r, dst54_r, dst21_r, dst43_r, dst65_r; v8i16 dst10_l, dst32_l, dst54_l, dst21_l, dst43_l, dst65_l; - v8i16 offset_vec, const_128, denom_vec; v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, weight_vec, rnd_vec; + v4i32 dst8, dst9, dst10, offset_vec; + v8i16 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + v8i16 zero = { 0 }; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); - SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + UNPCK_R_SB_SH(filter_vec, filter_vec); + SPLATI_W2_SH(filter_vec, 0, filt0, filt1); filter_vec = LD_SH(filter_y); UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask0 = LD_SB(ff_hevc_mask_arr); @@ -5065,43 +5988,61 @@ static void hevc_hv_uniwgt_4t_12w_msa(const uint8_t *src, weight_vec = __msa_fill_w(weight); rnd_vec = __msa_fill_w(rnd_val); - - offset_vec = __msa_fill_h(offset); - denom_vec = __msa_fill_h(rnd_val - 6); - const_128 = __msa_fill_h((128 * weight)); - offset_vec += __msa_srar_h(const_128, denom_vec); + offset_vec = __msa_fill_w(offset); src_tmp = src; dst_tmp = dst; LD_SB3(src_tmp, src_stride, src0, src1, src2); src_tmp += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - dsth0 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dsth1 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dsth2 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - ILVRL_H2_SH(dsth1, dsth0, dst10_r, dst10_l); - ILVRL_H2_SH(dsth2, dsth1, dst21_r, dst21_l); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dsth0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dsth1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dsth2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dsth3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + dsth4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dsth5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + ILVEV_H2_SH(dsth0, dsth2, dsth1, dsth3, dst10_r, dst10_l); + ILVEV_H2_SH(dsth2, dsth4, dsth3, dsth5, dst21_r, dst21_l); for (loop_cnt = 4; loop_cnt--;) { LD_SB4(src_tmp, src_stride, src3, src4, src5, src6); src_tmp += (4 * src_stride); - XORI_B4_128_SB(src3, src4, src5, src6); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); - dsth3 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dsth4 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dsth5 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dsth6 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - ILVRL_H2_SH(dsth3, dsth2, dst32_r, dst32_l); - ILVRL_H2_SH(dsth4, dsth3, dst43_r, dst43_l); - ILVRL_H2_SH(dsth5, dsth4, dst54_r, dst54_l); - ILVRL_H2_SH(dsth6, dsth5, dst65_r, dst65_l); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dsth0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dsth1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dsth2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dsth3 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dsth4, dsth0, dsth5, dsth1, dst32_r, dst32_l); + ILVEV_H2_SH(dsth0, dsth2, dsth1, dsth3, dst43_r, dst43_l); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dsth4 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dsth5 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dsth6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dsth7 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVEV_H2_SH(dsth2, dsth4, dsth3, dsth5, dst54_r, dst54_l); + ILVEV_H2_SH(dsth4, dsth6, dsth5, dsth7, dst65_r, dst65_l); + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); @@ -5118,11 +6059,14 @@ static void hevc_hv_uniwgt_4t_12w_msa(const uint8_t *src, MUL2(dst2_l, weight_vec, dst3_l, weight_vec, dst2_l, dst3_l); SRAR_W4_SW(dst0_r, dst0_l, dst1_r, dst1_l, rnd_vec); SRAR_W4_SW(dst2_r, dst2_l, dst3_r, dst3_l, rnd_vec); + ADD2(dst0_r, offset_vec, dst1_r, offset_vec, dst0_r, dst1_r); + ADD2(dst2_r, offset_vec, dst3_r, offset_vec, dst2_r, dst3_r); + ADD2(dst0_l, offset_vec, dst1_l, offset_vec, dst0_l, dst1_l); + ADD2(dst2_l, offset_vec, dst3_l, offset_vec, dst2_l, dst3_l); + CLIP_SW8_0_255(dst0_r, dst0_l, dst1_r, dst1_l, + dst2_r, dst2_l, dst3_r, dst3_l); PCKEV_H4_SH(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, dst3_r, tmp0, tmp1, tmp2, tmp3); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_D4(out0, out1, 0, 1, 0, 1, dst_tmp, dst_stride); dst_tmp += (4 * dst_stride); @@ -5131,7 +6075,8 @@ static void hevc_hv_uniwgt_4t_12w_msa(const uint8_t *src, dst10_l = dst54_l; dst21_r = dst65_r; dst21_l = dst65_l; - dsth2 = dsth6; + dsth4 = dsth6; + dsth5 = dsth7; } src += 8; @@ -5142,33 +6087,49 @@ static void hevc_hv_uniwgt_4t_12w_msa(const uint8_t *src, LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); VSHF_B2_SB(src0, src1, src0, src1, mask2, mask3, vec0, vec1); VSHF_B2_SB(src1, src2, src1, src2, mask2, mask3, vec2, vec3); - dst10 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst21 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - ILVRL_H2_SH(dst21, dst10, dst10_r, dst21_r); - dst22 = (v8i16) __msa_splati_d((v2i64) dst21, 1); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVL_B2_SH(zero, vec2, zero, vec3, tmp4, tmp6); + dst0 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst1 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst2 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + + ILVEV_H2_SH(dst0, dst1, dst1, dst2, dst10_r, dst21_r); for (loop_cnt = 2; loop_cnt--;) { LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); src += (8 * src_stride); - XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); VSHF_B2_SB(src3, src7, src3, src7, mask2, mask3, vec0, vec1); VSHF_B2_SB(src4, src8, src4, src8, mask2, mask3, vec2, vec3); VSHF_B2_SB(src5, src9, src5, src9, mask2, mask3, vec4, vec5); VSHF_B2_SB(src6, src10, src6, src10, mask2, mask3, vec6, vec7); - dst73 = HEVC_FILT_4TAP_SH(vec0, vec1, filt0, filt1); - dst84 = HEVC_FILT_4TAP_SH(vec2, vec3, filt0, filt1); - dst95 = HEVC_FILT_4TAP_SH(vec4, vec5, filt0, filt1); - dst106 = HEVC_FILT_4TAP_SH(vec6, vec7, filt0, filt1); - dst32_r = __msa_ilvr_h(dst73, dst22); - ILVRL_H2_SH(dst84, dst73, dst43_r, dst87_r); - ILVRL_H2_SH(dst95, dst84, dst54_r, dst98_r); - ILVRL_H2_SH(dst106, dst95, dst65_r, dst109_r); - dst22 = (v8i16) __msa_splati_d((v2i64) dst73, 1); - dst76_r = __msa_ilvr_h(dst22, dst106); + + ILVRL_B2_SH(zero, vec0, tmp0, tmp1); + ILVRL_B2_SH(zero, vec1, tmp2, tmp3); + ILVRL_B2_SH(zero, vec2, tmp4, tmp5); + ILVRL_B2_SH(zero, vec3, tmp6, tmp7); + dst3 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst7 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst4 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst8 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + ILVRL_B2_SH(zero, vec4, tmp0, tmp1); + ILVRL_B2_SH(zero, vec5, tmp2, tmp3); + ILVRL_B2_SH(zero, vec6, tmp4, tmp5); + ILVRL_B2_SH(zero, vec7, tmp6, tmp7); + dst5 = HEVC_FILT_4TAP_SW(tmp0, tmp2, filt0, filt1); + dst9 = HEVC_FILT_4TAP_SW(tmp1, tmp3, filt0, filt1); + dst6 = HEVC_FILT_4TAP_SW(tmp4, tmp6, filt0, filt1); + dst10 = HEVC_FILT_4TAP_SW(tmp5, tmp7, filt0, filt1); + + dst32_r = __msa_ilvev_h(dst3, dst2); + ILVEV_H2_SH(dst3, dst4, dst7, dst8, dst43_r, dst87_r); + ILVEV_H2_SH(dst4, dst5, dst8, dst9, dst54_r, dst98_r); + ILVEV_H2_SH(dst5, dst6, dst9, dst10, dst65_r, dst109_r); + dst76_r = __msa_ilvev_h(dst7, dst6); dst0 = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst1 = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); dst2 = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); @@ -5185,18 +6146,21 @@ static void hevc_hv_uniwgt_4t_12w_msa(const uint8_t *src, MUL2(dst6, weight_vec, dst7, weight_vec, dst6, dst7); SRAR_W4_SW(dst0, dst1, dst2, dst3, rnd_vec); SRAR_W4_SW(dst4, dst5, dst6, dst7, rnd_vec); + ADD2(dst0, offset_vec, dst1, offset_vec, dst0, dst1); + ADD2(dst2, offset_vec, dst3, offset_vec, dst2, dst3); + ADD2(dst4, offset_vec, dst5, offset_vec, dst4, dst5); + ADD2(dst6, offset_vec, dst7, offset_vec, dst6, dst7); + CLIP_SW8_0_255(dst0, dst1, dst2, dst3, + dst4, dst5, dst6, dst7); PCKEV_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, tmp0, tmp1, tmp2, tmp3); - ADD2(tmp0, offset_vec, tmp1, offset_vec, tmp0, tmp1); - ADD2(tmp2, offset_vec, tmp3, offset_vec, tmp2, tmp3); - CLIP_SH4_0_255(tmp0, tmp1, tmp2, tmp3); PCKEV_B2_UB(tmp1, tmp0, tmp3, tmp2, out0, out1); ST_W8(out0, out1, 0, 1, 2, 3, 0, 1, 2, 3, dst, dst_stride); dst += (8 * dst_stride); dst10_r = dst98_r; dst21_r = dst109_r; - dst22 = (v8i16) __msa_splati_d((v2i64) dst106, 1); + dst2 = dst10; } } diff --git a/libavcodec/mips/hevcdsp_msa.c b/libavcodec/mips/hevcdsp_msa.c index 8841a31369..9c12029c1f 100644 --- a/libavcodec/mips/hevcdsp_msa.c +++ b/libavcodec/mips/hevcdsp_msa.c @@ -81,12 +81,13 @@ static void hevc_copy_6w_msa(const uint8_t *src, int32_t src_stride, int16_t *dst, int32_t dst_stride, int32_t height) { - uint32_t loop_cnt; + uint32_t loop_cnt = (height >> 3); + uint32_t res = height & 0x07; v16i8 zero = { 0 }; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v8i16 in0, in1, in2, in3, in4, in5, in6, in7; - for (loop_cnt = (height >> 3); loop_cnt--;) { + for (; loop_cnt--; ) { LD_SB8(src, src_stride, src0, src1, src2, src3, src4, src5, src6, src7); src += (8 * src_stride); @@ -99,6 +100,19 @@ static void hevc_copy_6w_msa(const uint8_t *src, int32_t src_stride, ST12x8_UB(in0, in1, in2, in3, in4, in5, in6, in7, dst, 2 * dst_stride); dst += (8 * dst_stride); } + for (; res--; ) { + uint64_t out0; + uint32_t out1; + src0 = LD_SB(src); + src += src_stride; + in0 = (v8i16)__msa_ilvr_b((v16i8) zero, (v16i8) src0); + in0 = in0 << 6; + out0 = __msa_copy_u_d((v2i64) in0, 0); + out1 = __msa_copy_u_w((v4i32) in0, 2); + SD(out0, dst); + SW(out1, dst + 4); + dst += dst_stride; + } } static void hevc_copy_8w_msa(const uint8_t *src, int32_t src_stride, @@ -167,6 +181,7 @@ static void hevc_copy_12w_msa(const uint8_t *src, int32_t src_stride, int32_t height) { uint32_t loop_cnt; + uint32_t res = height & 0x07; v16i8 zero = { 0 }; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v8i16 in0, in1, in0_r, in1_r, in2_r, in3_r; @@ -197,6 +212,19 @@ static void hevc_copy_12w_msa(const uint8_t *src, int32_t src_stride, ST_D4(in0, in1, 0, 1, 0, 1, dst + 8, dst_stride); dst += (4 * dst_stride); } + for (; res--; ) { + uint64_t out0; + src0 = LD_SB(src); + src += src_stride; + in0_r = (v8i16)__msa_ilvr_b((v16i8) zero, (v16i8) src0); + in0 = (v8i16)__msa_ilvl_b((v16i8) zero, (v16i8) src0); + in0_r = in0_r << 6; + in0 = in0 << 6; + ST_UH(in0_r, dst); + out0 = __msa_copy_u_d((v2i64) in0, 0); + SD(out0, dst + 8); + dst += dst_stride; + } } static void hevc_copy_16w_msa(const uint8_t *src, int32_t src_stride, @@ -450,6 +478,7 @@ static void hevc_hz_8t_4w_msa(const uint8_t *src, int32_t src_stride, const int8_t *filter, int32_t height) { uint32_t loop_cnt; + uint32_t res = (height & 0x07) >> 1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v8i16 filt0, filt1, filt2, filt3; v16i8 mask1, mask2, mask3; @@ -498,6 +527,18 @@ static void hevc_hz_8t_4w_msa(const uint8_t *src, int32_t src_stride, ST_D8(dst0, dst1, dst2, dst3, 0, 1, 0, 1, 0, 1, 0, 1, dst, dst_stride); dst += (8 * dst_stride); } + for (; res--; ) { + LD_SB2(src, src_stride, src0, src1); + src += 2 * src_stride; + XORI_B2_128_SB(src0, src1); + VSHF_B4_SB(src0, src1, mask0, mask1, mask2, mask3, + vec0, vec1, vec2, vec3); + dst0 = const_vec; + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, + dst0, dst0, dst0, dst0); + ST_D2(dst0, 0, 1, dst, dst_stride); + dst += 2 * dst_stride; + } } static void hevc_hz_8t_8w_msa(const uint8_t *src, int32_t src_stride, @@ -992,6 +1033,7 @@ static void hevc_vt_8t_4w_msa(const uint8_t *src, int32_t src_stride, const int8_t *filter, int32_t height) { int32_t loop_cnt; + int32_t res = (height & 0x07) >> 1; v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; v16i8 src9, src10, src11, src12, src13, src14; v16i8 src10_r, src32_r, src54_r, src76_r, src98_r; @@ -1055,6 +1097,22 @@ static void hevc_vt_8t_4w_msa(const uint8_t *src, int32_t src_stride, src6554 = src14131312; src6 = src14; } + for (; res--; ) { + LD_SB2(src, src_stride, src7, src8); + src += 2 * src_stride; + ILVR_B2_SB(src7, src6, src8, src7, src76_r, src87_r); + src8776 = (v16i8)__msa_ilvr_d((v2i64) src87_r, src76_r); + src8776 = (v16i8)__msa_xori_b((v16i8) src8776, 128); + dst10 = const_vec; + DPADD_SB4_SH(src2110, src4332, src6554, src8776, + filt0, filt1, filt2, filt3, dst10, dst10, dst10, dst10); + ST_D2(dst10, 0, 1, dst, dst_stride); + dst += 2 * dst_stride; + src2110 = src4332; + src4332 = src6554; + src6554 = src8776; + src6 = src8; + } } static void hevc_vt_8t_8w_msa(const uint8_t *src, int32_t src_stride, @@ -2661,6 +2719,7 @@ static void hevc_vt_4t_6w_msa(const uint8_t *src, int32_t height) { int32_t loop_cnt; + int32_t res = height & 0x03; uint32_t dst_val_int0, dst_val_int1, dst_val_int2, dst_val_int3; uint64_t dst_val0, dst_val1, dst_val2, dst_val3; v16i8 src0, src1, src2, src3, src4; @@ -2725,6 +2784,29 @@ static void hevc_vt_4t_6w_msa(const uint8_t *src, SW(dst_val_int3, dst + 4); dst += dst_stride; } + if (res) { + LD_SB2(src, src_stride, src3, src4); + XORI_B2_128_SB(src3, src4); + ILVR_B2_SB(src3, src2, src4, src3, src32_r, src43_r); + + dst0_r = const_vec; + DPADD_SB2_SH(src10_r, src32_r, filt0, filt1, dst0_r, dst0_r); + dst1_r = const_vec; + DPADD_SB2_SH(src21_r, src43_r, filt0, filt1, dst1_r, dst1_r); + + dst_val0 = __msa_copy_u_d((v2i64) dst0_r, 0); + dst_val1 = __msa_copy_u_d((v2i64) dst1_r, 0); + + dst_val_int0 = __msa_copy_u_w((v4i32) dst0_r, 2); + dst_val_int1 = __msa_copy_u_w((v4i32) dst1_r, 2); + + SD(dst_val0, dst); + SW(dst_val_int0, dst + 4); + dst += dst_stride; + SD(dst_val1, dst); + SW(dst_val_int1, dst + 4); + dst += dst_stride; + } } static void hevc_vt_4t_8x2_msa(const uint8_t *src,