From patchwork Mon Nov 6 12:36:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: kaustubh.raste@imgtec.com X-Patchwork-Id: 5886 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.90 with SMTP id m26csp2841057jah; Mon, 6 Nov 2017 04:36:50 -0800 (PST) X-Google-Smtp-Source: ABhQp+QWVVvdtghj13dQ4Vg2oc/yR1uLZLka3AB+3MHZKfnSAqIbdWqOZrZ4RgpABiL/uM5+MmBa X-Received: by 10.28.160.21 with SMTP id j21mr3306849wme.15.1509971810625; Mon, 06 Nov 2017 04:36:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1509971810; cv=none; d=google.com; s=arc-20160816; b=vQdqWgzNVTQTzZltJyN9CikoepO2TB1OQpHz9cC0KDu+Ghlie5tIBPbsag9wlDLpdB vB8/VOGqRm+GZd/iBB/JSQeBx90lFTfMRMBIOhcy5IIKWl7wYICx9agy/UnRe205W78e GbKNY520iBM+CG7RInFhx2rZGENjX9LmKrJObmCAdOGDVC7sA3acrSIbIbOOfoc0CV6a iGyQhIs/A70CYd0aB0pn6jucyRlmaEjgzDZ/E1SdErY4tt9hkgJJdcnAU+9Lv+sjlCdT cLUy2hjpKNWh0gZ+hSKyr1i8EuFR3JttAe+nea1V+T+AwN6b4ALd/84fuviKdwXQQCWL FTSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :delivered-to:arc-authentication-results; bh=jRva4cqL4TnkJvy5hNwCOCFVha2eo79Cmzr2igvEK1Y=; b=ELXNmUe+/wGdtRUFfFu7aOz6P+YrrYqXoUp6ljT7Z6uAB+2DJC1t8TAB+msPDdELEN 24XmWmp/FD/ltZtLR2+OGzD4ZB8oBdjoE+oG4TPkcb3KM9e0RcSq9BBoupKY3PB5k2J4 C/zU4PgnsRoUZR8eQMQ8Rz7tIiDt3KgSxmjsnBF9VM9J33IQ6iu2IFfzHkhgU5p4Ox6q O9Z3JUcetgbmvyahVZ9xr8/bGA2mXuRyHBUMqiaarR5RRj+29okSSFbm72fe4AoqbmhE s5ZOl2o9XxQ/+kqmHUHhKPkmaIWkvFsXUCKoC68UycTe+ywXhnMYGRgTJriuuugnhihl LkFw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n16si2407044wrn.262.2017.11.06.04.36.49; Mon, 06 Nov 2017 04:36:50 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 86594689AC6; Mon, 6 Nov 2017 14:36:36 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mailapp01.imgtec.com (mailapp01.imgtec.com [195.59.15.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A4A106897FF for ; Mon, 6 Nov 2017 14:36:30 +0200 (EET) Received: from hhmail02.hh.imgtec.org (unknown [10.100.10.20]) by Forcepoint Email with ESMTPS id A875EDDB75184 for ; Mon, 6 Nov 2017 12:36:37 +0000 (GMT) Received: from pudesk204.pu.imgtec.org (192.168.91.13) by hhmail02.hh.imgtec.org (10.100.10.20) with Microsoft SMTP Server (TLS) id 14.3.361.1; Mon, 6 Nov 2017 12:36:39 +0000 From: To: Date: Mon, 6 Nov 2017 18:06:53 +0530 Message-ID: <1509971813-28930-1-git-send-email-kaustubh.raste@imgtec.com> X-Mailer: git-send-email 1.7.9.5 MIME-Version: 1.0 X-Originating-IP: [192.168.91.13] Subject: [FFmpeg-devel] [PATCH] avcodec/mips: Improve hevc non-uni hv mc msa functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Kaustubh Raste Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Kaustubh Raste Use mask buffer. Signed-off-by: Kaustubh Raste --- libavcodec/mips/hevcdsp_msa.c | 1478 ++++++++++++++++++++++++----------------- 1 file changed, 870 insertions(+), 608 deletions(-) diff --git a/libavcodec/mips/hevcdsp_msa.c b/libavcodec/mips/hevcdsp_msa.c index 73cc3ea..b17127c 100644 --- a/libavcodec/mips/hevcdsp_msa.c +++ b/libavcodec/mips/hevcdsp_msa.c @@ -22,6 +22,13 @@ #include "libavcodec/mips/hevcdsp_mips.h" #include "libavcodec/mips/hevc_macros_msa.h" +static const uint8_t ff_hevc_mask_arr[16 * 2] __attribute__((aligned(0x40))) = { + /* 8 width cases */ + 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, + /* 4 width cases */ + 0, 1, 1, 2, 2, 3, 3, 4, 16, 17, 17, 18, 18, 19, 19, 20 +}; + static void hevc_copy_4w_msa(uint8_t *src, int32_t src_stride, int16_t *dst, int32_t dst_stride, int32_t height) @@ -1308,31 +1315,28 @@ static void hevc_hv_8t_4w_msa(uint8_t *src, int32_t src_stride, int32_t height) { uint32_t loop_cnt; - v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; + int32_t dst_stride_in_bytes = 2 * dst_stride; + v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v8i16 filt0, filt1, filt2, filt3; - v4i32 filt_h0, filt_h1, filt_h2, filt_h3; + v8i16 filt_h0, filt_h1, filt_h2, filt_h3; v16i8 mask1, mask2, mask3; v8i16 filter_vec, const_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst30, dst41, dst52, dst63, dst66, dst87; - v4i32 dst0_r, dst1_r; - v8i16 dst10_r, dst32_r, dst54_r, dst76_r; - v8i16 dst21_r, dst43_r, dst65_r, dst87_r; - v16i8 mask0 = { - 0, 1, 1, 2, 2, 3, 3, 4, 16, 17, 17, 18, 18, 19, 19, 20 - }; - v8u16 mask4 = { 0, 4, 1, 5, 2, 6, 3, 7 }; + v8i16 dst30, dst41, dst52, dst63, dst66, dst97, dst108; + v4i32 dst0_r, dst1_r, dst2_r, dst3_r; + v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst98_r; + v8i16 dst21_r, dst43_r, dst65_r, dst87_r, dst109_r; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); src -= ((3 * src_stride) + 3); filter_vec = LD_SH(filter_x); SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W4_SW(filter_vec, filt_h0, filt_h1, filt_h2, filt_h3); + SPLATI_W4_SH(filter_vec, filt_h0, filt_h1, filt_h2, filt_h3); mask1 = mask0 + 2; mask2 = mask0 + 4; @@ -1364,47 +1368,56 @@ static void hevc_hv_8t_4w_msa(uint8_t *src, int32_t src_stride, DPADD_SB4_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, filt3, dst63, dst63, dst63, dst63); - ILVR_H3_SH(dst41, dst30, dst52, dst41, dst63, dst52, - dst10_r, dst21_r, dst32_r); - dst43_r = __msa_ilvl_h(dst41, dst30); - dst54_r = __msa_ilvl_h(dst52, dst41); - dst65_r = __msa_ilvl_h(dst63, dst52); + ILVRL_H2_SH(dst41, dst30, dst10_r, dst43_r); + ILVRL_H2_SH(dst52, dst41, dst21_r, dst54_r); + ILVRL_H2_SH(dst63, dst52, dst32_r, dst65_r); dst66 = (v8i16) __msa_splati_d((v2i64) dst63, 1); - for (loop_cnt = height >> 1; loop_cnt--;) { - LD_SB2(src, src_stride, src7, src8); - src += (2 * src_stride); - XORI_B2_128_SB(src7, src8); + for (loop_cnt = height >> 2; loop_cnt--;) { + LD_SB4(src, src_stride, src7, src8, src9, src10); + src += (4 * src_stride); + XORI_B4_128_SB(src7, src8, src9, src10); - VSHF_B4_SB(src7, src8, mask0, mask1, mask2, mask3, + VSHF_B4_SB(src7, src9, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); - dst87 = const_vec; + VSHF_B4_SB(src8, src10, mask0, mask1, mask2, mask3, + vec4, vec5, vec6, vec7); + dst97 = const_vec; + dst108 = const_vec; DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, - dst87, dst87, dst87, dst87); - dst76_r = __msa_ilvr_h(dst87, dst66); + dst97, dst97, dst97, dst97); + DPADD_SB4_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, filt3, + dst108, dst108, dst108, dst108); + + dst76_r = __msa_ilvr_h(dst97, dst66); + ILVRL_H2_SH(dst108, dst97, dst87_r, dst109_r); + dst66 = (v8i16) __msa_splati_d((v2i64) dst97, 1); + dst98_r = __msa_ilvr_h(dst66, dst108); + dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, filt_h1, filt_h2, filt_h3); - dst87_r = __msa_vshf_h((v8i16) mask4, dst87, dst87); dst1_r = HEVC_FILT_8TAP(dst21_r, dst43_r, dst65_r, dst87_r, filt_h0, filt_h1, filt_h2, filt_h3); - dst0_r >>= 6; - dst1_r >>= 6; - - dst0_r = (v4i32) __msa_pckev_h((v8i16) dst1_r, (v8i16) dst0_r); - ST8x2_UB(dst0_r, dst, (2 * dst_stride)); - dst += (2 * dst_stride); + dst2_r = HEVC_FILT_8TAP(dst32_r, dst54_r, dst76_r, dst98_r, + filt_h0, filt_h1, filt_h2, filt_h3); + dst3_r = HEVC_FILT_8TAP(dst43_r, dst65_r, dst87_r, dst109_r, + filt_h0, filt_h1, filt_h2, filt_h3); + SRA_4V(dst0_r, dst1_r, dst2_r, dst3_r, 6); + PCKEV_H2_SW(dst1_r, dst0_r, dst3_r, dst2_r, dst0_r, dst2_r); + ST8x4_UB(dst0_r, dst2_r, dst, dst_stride_in_bytes); + dst += (4 * dst_stride); - dst10_r = dst32_r; - dst32_r = dst54_r; - dst54_r = dst76_r; - dst21_r = dst43_r; - dst43_r = dst65_r; - dst65_r = dst87_r; - dst66 = (v8i16) __msa_splati_d((v2i64) dst87, 1); + dst10_r = dst54_r; + dst32_r = dst76_r; + dst54_r = dst98_r; + dst21_r = dst65_r; + dst43_r = dst87_r; + dst65_r = dst109_r; + dst66 = (v8i16) __msa_splati_d((v2i64) dst108, 1); } } -static void hevc_hv_8t_8multx2mult_msa(uint8_t *src, +static void hevc_hv_8t_8multx1mult_msa(uint8_t *src, int32_t src_stride, int16_t *dst, int32_t dst_stride, @@ -1415,19 +1428,17 @@ static void hevc_hv_8t_8multx2mult_msa(uint8_t *src, uint32_t loop_cnt, cnt; uint8_t *src_tmp; int16_t *dst_tmp; - v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; + v16i8 src0, src1, src2, src3, src4, src5, src6, src7; v8i16 filt0, filt1, filt2, filt3; - v4i32 filt_h0, filt_h1, filt_h2, filt_h3; + v8i16 filt_h0, filt_h1, filt_h2, filt_h3; v16i8 mask1, mask2, mask3; v8i16 filter_vec, const_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8; + v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; v4i32 dst0_r, dst0_l; v8i16 dst10_r, dst32_r, dst54_r, dst76_r; v8i16 dst10_l, dst32_l, dst54_l, dst76_l; - v8i16 dst21_r, dst43_r, dst65_r, dst87_r; - v8i16 dst21_l, dst43_l, dst65_l, dst87_l; v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; src -= ((3 * src_stride) + 3); @@ -1435,10 +1446,9 @@ static void hevc_hv_8t_8multx2mult_msa(uint8_t *src, SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W4_SW(filter_vec, filt_h0, filt_h1, filt_h2, filt_h3); + SPLATI_W4_SH(filter_vec, filt_h0, filt_h1, filt_h2, filt_h3); mask1 = mask0 + 2; mask2 = mask0 + 4; @@ -1494,17 +1504,10 @@ static void hevc_hv_8t_8multx2mult_msa(uint8_t *src, DPADD_SB4_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, filt3, dst6, dst6, dst6, dst6); - ILVR_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst2, dst1, - dst10_r, dst32_r, dst54_r, dst21_r); - ILVR_H2_SH(dst4, dst3, dst6, dst5, dst43_r, dst65_r); - ILVL_H4_SH(dst1, dst0, dst3, dst2, dst5, dst4, dst2, dst1, - dst10_l, dst32_l, dst54_l, dst21_l); - ILVL_H2_SH(dst4, dst3, dst6, dst5, dst43_l, dst65_l); - - for (loop_cnt = height >> 1; loop_cnt--;) { - LD_SB2(src_tmp, src_stride, src7, src8); - XORI_B2_128_SB(src7, src8); - src_tmp += 2 * src_stride; + for (loop_cnt = height; loop_cnt--;) { + src7 = LD_SB(src_tmp); + src7 = (v16i8) __msa_xori_b((v16u8) src7, 128); + src_tmp += src_stride; VSHF_B4_SB(src7, src7, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); @@ -1512,6 +1515,9 @@ static void hevc_hv_8t_8multx2mult_msa(uint8_t *src, DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, dst7, dst7, dst7, dst7); + ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); + ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); + ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, filt_h1, filt_h2, filt_h3); @@ -1524,37 +1530,13 @@ static void hevc_hv_8t_8multx2mult_msa(uint8_t *src, ST_SW(dst0_r, dst_tmp); dst_tmp += dst_stride; - VSHF_B4_SB(src8, src8, mask0, mask1, mask2, mask3, - vec0, vec1, vec2, vec3); - dst8 = const_vec; - DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, - dst8, dst8, dst8, dst8); - - ILVRL_H2_SH(dst8, dst7, dst87_r, dst87_l); - dst6 = dst8; - dst0_r = HEVC_FILT_8TAP(dst21_r, dst43_r, dst65_r, dst87_r, - filt_h0, filt_h1, filt_h2, filt_h3); - dst0_l = HEVC_FILT_8TAP(dst21_l, dst43_l, dst65_l, dst87_l, - filt_h0, filt_h1, filt_h2, filt_h3); - dst0_r >>= 6; - dst0_l >>= 6; - - dst0_r = (v4i32) __msa_pckev_h((v8i16) dst0_l, (v8i16) dst0_r); - ST_SW(dst0_r, dst_tmp); - dst_tmp += dst_stride; - - dst10_r = dst32_r; - dst32_r = dst54_r; - dst54_r = dst76_r; - dst10_l = dst32_l; - dst32_l = dst54_l; - dst54_l = dst76_l; - dst21_r = dst43_r; - dst43_r = dst65_r; - dst65_r = dst87_r; - dst21_l = dst43_l; - dst43_l = dst65_l; - dst65_l = dst87_l; + dst0 = dst1; + dst1 = dst2; + dst2 = dst3; + dst3 = dst4; + dst4 = dst5; + dst5 = dst6; + dst6 = dst7; } src += 8; @@ -1567,7 +1549,7 @@ static void hevc_hv_8t_8w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, + hevc_hv_8t_8multx1mult_msa(src, src_stride, dst, dst_stride, filter_x, filter_y, height, 8); } @@ -1576,11 +1558,195 @@ static void hevc_hv_8t_12w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height, 8); + uint32_t loop_cnt; + int32_t dst_stride_in_bytes = 2 * dst_stride; + uint8_t *src_tmp; + int16_t *dst_tmp; + v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; + v16i8 mask0, mask1, mask2, mask3, mask4, mask5, mask6, mask7; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v16i8 vec8, vec9, vec10, vec11, vec12, vec13, vec14, vec15; + v8i16 filt0, filt1, filt2, filt3, filt_h0, filt_h1, filt_h2, filt_h3; + v8i16 filter_vec, const_vec; + v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; + v8i16 dst30, dst41, dst52, dst63, dst66, dst97, dst108; + v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst98_r, dst21_r, dst43_r; + v8i16 dst65_r, dst87_r, dst109_r, dst10_l, dst32_l, dst54_l, dst76_l; + v4i32 dst0_r, dst0_l, dst1_r, dst2_r, dst3_r; + + src -= ((3 * src_stride) + 3); + filter_vec = LD_SH(filter_x); + SPLATI_H4_SH(filter_vec, 0, 1, 2, 3, filt0, filt1, filt2, filt3); + + filter_vec = LD_SH(filter_y); + UNPCK_R_SB_SH(filter_vec, filter_vec); + + SPLATI_W4_SH(filter_vec, filt_h0, filt_h1, filt_h2, filt_h3); + + mask0 = LD_SB(ff_hevc_mask_arr); + mask1 = mask0 + 2; + mask2 = mask0 + 4; + mask3 = mask0 + 6; + + const_vec = __msa_ldi_h(128); + const_vec <<= 6; + + src_tmp = src; + dst_tmp = dst; + + LD_SB7(src_tmp, src_stride, src0, src1, src2, src3, src4, src5, src6); + src_tmp += (7 * src_stride); + XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); + + /* row 0 row 1 row 2 row 3 */ + VSHF_B4_SB(src0, src0, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); + VSHF_B4_SB(src1, src1, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); + VSHF_B4_SB(src2, src2, mask0, mask1, mask2, mask3, vec8, vec9, vec10, + vec11); + VSHF_B4_SB(src3, src3, mask0, mask1, mask2, mask3, vec12, vec13, vec14, + vec15); + dst0 = const_vec; + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, dst0, dst0, + dst0, dst0); + dst1 = const_vec; + DPADD_SB4_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, filt3, dst1, dst1, + dst1, dst1); + dst2 = const_vec; + DPADD_SB4_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, filt3, dst2, + dst2, dst2, dst2); + dst3 = const_vec; + DPADD_SB4_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, filt3, dst3, + dst3, dst3, dst3); + + /* row 4 row 5 row 6 */ + VSHF_B4_SB(src4, src4, mask0, mask1, mask2, mask3, vec0, vec1, vec2, vec3); + VSHF_B4_SB(src5, src5, mask0, mask1, mask2, mask3, vec4, vec5, vec6, vec7); + VSHF_B4_SB(src6, src6, mask0, mask1, mask2, mask3, vec8, vec9, vec10, + vec11); + dst4 = const_vec; + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, dst4, dst4, + dst4, dst4); + dst5 = const_vec; + DPADD_SB4_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, filt3, dst5, dst5, + dst5, dst5); + dst6 = const_vec; + DPADD_SB4_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, filt3, dst6, + dst6, dst6, dst6); + + for (loop_cnt = height; loop_cnt--;) { + src7 = LD_SB(src_tmp); + src7 = (v16i8) __msa_xori_b((v16u8) src7, 128); + src_tmp += src_stride; + + VSHF_B4_SB(src7, src7, mask0, mask1, mask2, mask3, vec0, vec1, vec2, + vec3); + dst7 = const_vec; + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, dst7, + dst7, dst7, dst7); + + ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); + ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); + ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); + ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); + dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst0_l = HEVC_FILT_8TAP(dst10_l, dst32_l, dst54_l, dst76_l, filt_h0, + filt_h1, filt_h2, filt_h3); + dst0_r >>= 6; + dst0_l >>= 6; + + dst0_r = (v4i32) __msa_pckev_h((v8i16) dst0_l, (v8i16) dst0_r); + ST_SW(dst0_r, dst_tmp); + dst_tmp += dst_stride; + + dst0 = dst1; + dst1 = dst2; + dst2 = dst3; + dst3 = dst4; + dst4 = dst5; + dst5 = dst6; + dst6 = dst7; + } + + src += 8; + dst += 8; + + mask4 = LD_SB(ff_hevc_mask_arr + 16); + mask5 = mask4 + 2; + mask6 = mask4 + 4; + mask7 = mask4 + 6; + + LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); + src += (7 * src_stride); + XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); + + VSHF_B4_SB(src0, src3, mask4, mask5, mask6, mask7, vec0, vec1, vec2, vec3); + VSHF_B4_SB(src1, src4, mask4, mask5, mask6, mask7, vec4, vec5, vec6, vec7); + VSHF_B4_SB(src2, src5, mask4, mask5, mask6, mask7, vec8, vec9, vec10, + vec11); + VSHF_B4_SB(src3, src6, mask4, mask5, mask6, mask7, vec12, vec13, vec14, + vec15); + dst30 = const_vec; + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, dst30, + dst30, dst30, dst30); + dst41 = const_vec; + DPADD_SB4_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, filt3, dst41, + dst41, dst41, dst41); + dst52 = const_vec; + DPADD_SB4_SH(vec8, vec9, vec10, vec11, filt0, filt1, filt2, filt3, dst52, + dst52, dst52, dst52); + dst63 = const_vec; + DPADD_SB4_SH(vec12, vec13, vec14, vec15, filt0, filt1, filt2, filt3, dst63, + dst63, dst63, dst63); + + ILVRL_H2_SH(dst41, dst30, dst10_r, dst43_r); + ILVRL_H2_SH(dst52, dst41, dst21_r, dst54_r); + ILVRL_H2_SH(dst63, dst52, dst32_r, dst65_r); - hevc_hv_8t_4w_msa(src + 8, src_stride, dst + 8, dst_stride, - filter_x, filter_y, height); + dst66 = (v8i16) __msa_splati_d((v2i64) dst63, 1); + + for (loop_cnt = height >> 2; loop_cnt--;) { + LD_SB4(src, src_stride, src7, src8, src9, src10); + src += (4 * src_stride); + XORI_B4_128_SB(src7, src8, src9, src10); + + VSHF_B4_SB(src7, src9, mask4, mask5, mask6, mask7, vec0, vec1, vec2, + vec3); + VSHF_B4_SB(src8, src10, mask4, mask5, mask6, mask7, vec4, vec5, vec6, + vec7); + dst97 = const_vec; + dst108 = const_vec; + DPADD_SB4_SH(vec0, vec1, vec2, vec3, filt0, filt1, filt2, filt3, dst97, + dst97, dst97, dst97); + DPADD_SB4_SH(vec4, vec5, vec6, vec7, filt0, filt1, filt2, filt3, dst108, + dst108, dst108, dst108); + + dst76_r = __msa_ilvr_h(dst97, dst66); + ILVRL_H2_SH(dst108, dst97, dst87_r, dst109_r); + dst66 = (v8i16) __msa_splati_d((v2i64) dst97, 1); + dst98_r = __msa_ilvr_h(dst66, dst108); + + dst0_r = HEVC_FILT_8TAP(dst10_r, dst32_r, dst54_r, dst76_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst1_r = HEVC_FILT_8TAP(dst21_r, dst43_r, dst65_r, dst87_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst2_r = HEVC_FILT_8TAP(dst32_r, dst54_r, dst76_r, dst98_r, filt_h0, + filt_h1, filt_h2, filt_h3); + dst3_r = HEVC_FILT_8TAP(dst43_r, dst65_r, dst87_r, dst109_r, filt_h0, + filt_h1, filt_h2, filt_h3); + SRA_4V(dst0_r, dst1_r, dst2_r, dst3_r, 6); + PCKEV_H2_SW(dst1_r, dst0_r, dst3_r, dst2_r, dst0_r, dst2_r); + ST8x4_UB(dst0_r, dst2_r, dst, dst_stride_in_bytes); + dst += (4 * dst_stride); + + dst10_r = dst54_r; + dst32_r = dst76_r; + dst54_r = dst98_r; + dst21_r = dst65_r; + dst43_r = dst87_r; + dst65_r = dst109_r; + dst66 = (v8i16) __msa_splati_d((v2i64) dst108, 1); + } } static void hevc_hv_8t_16w_msa(uint8_t *src, int32_t src_stride, @@ -1588,7 +1754,7 @@ static void hevc_hv_8t_16w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, + hevc_hv_8t_8multx1mult_msa(src, src_stride, dst, dst_stride, filter_x, filter_y, height, 16); } @@ -1597,7 +1763,7 @@ static void hevc_hv_8t_24w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, + hevc_hv_8t_8multx1mult_msa(src, src_stride, dst, dst_stride, filter_x, filter_y, height, 24); } @@ -1606,7 +1772,7 @@ static void hevc_hv_8t_32w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, + hevc_hv_8t_8multx1mult_msa(src, src_stride, dst, dst_stride, filter_x, filter_y, height, 32); } @@ -1615,7 +1781,7 @@ static void hevc_hv_8t_48w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, + hevc_hv_8t_8multx1mult_msa(src, src_stride, dst, dst_stride, filter_x, filter_y, height, 48); } @@ -1624,7 +1790,7 @@ static void hevc_hv_8t_64w_msa(uint8_t *src, int32_t src_stride, const int8_t *filter_x, const int8_t *filter_y, int32_t height) { - hevc_hv_8t_8multx2mult_msa(src, src_stride, dst, dst_stride, + hevc_hv_8t_8multx1mult_msa(src, src_stride, dst, dst_stride, filter_x, filter_y, height, 64); } @@ -2994,69 +3160,52 @@ static void hevc_hv_4t_4x2_msa(uint8_t *src, const int8_t *filter_x, const int8_t *filter_y) { + int32_t dst_stride_in_bytes = 2 * dst_stride; v16i8 src0, src1, src2, src3, src4; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); v16i8 mask1; v8i16 filter_vec, const_vec; v16i8 vec0, vec1, vec2, vec3, vec4, vec5; - v8i16 dst0, dst1, dst2, dst3, dst4; - v4i32 dst0_r, dst1_r; - v8i16 dst10_r, dst32_r, dst21_r, dst43_r; + v8i16 dst20, dst31, dst42, dst10, dst32, dst21, dst43; + v4i32 dst0, dst1; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; const_vec = __msa_ldi_h(128); const_vec <<= 6; - LD_SB3(src, src_stride, src0, src1, src2); - src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); - - VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); - VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); - VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - - dst0 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); - dst1 = const_vec; - DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); - dst2 = const_vec; - DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); - ILVR_H2_SH(dst1, dst0, dst2, dst1, dst10_r, dst21_r); - - LD_SB2(src, src_stride, src3, src4); - XORI_B2_128_SB(src3, src4); - - VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); - dst3 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); - - dst32_r = __msa_ilvr_h(dst3, dst2); - dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); - dst0_r >>= 6; - - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - dst43_r = __msa_ilvr_h(dst4, dst3); - dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); - dst1_r >>= 6; - - dst0_r = (v4i32) __msa_pckev_h((v8i16) dst1_r, (v8i16) dst0_r); - ST8x2_UB(dst0_r, dst, 2 * dst_stride); + LD_SB5(src, src_stride, src0, src1, src2, src3, src4); + XORI_B5_128_SB(src0, src1, src2, src3, src4); + VSHF_B2_SB(src0, src2, src0, src2, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src1, src3, src1, src3, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src2, src4, src2, src4, mask0, mask1, vec4, vec5); + + dst20 = const_vec; + dst31 = const_vec; + dst42 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst20, dst20); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst31, dst31); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst42, dst42); + ILVRL_H2_SH(dst31, dst20, dst10, dst32); + ILVRL_H2_SH(dst42, dst31, dst21, dst43); + + dst0 = HEVC_FILT_4TAP(dst10, dst32, filt_h0, filt_h1); + dst1 = HEVC_FILT_4TAP(dst21, dst43, filt_h0, filt_h1); + dst0 >>= 6; + dst1 >>= 6; + dst0 = (v4i32) __msa_pckev_h((v8i16) dst1, (v8i16) dst0); + ST8x2_UB(dst0, dst, dst_stride_in_bytes); } static void hevc_hv_4t_4x4_msa(uint8_t *src, @@ -3066,16 +3215,16 @@ static void hevc_hv_4t_4x4_msa(uint8_t *src, const int8_t *filter_x, const int8_t *filter_y) { + int32_t dst_stride_in_bytes = 2 * dst_stride; v16i8 src0, src1, src2, src3, src4, src5, src6; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); v16i8 mask1; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; v8i16 filter_vec, const_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v4i32 dst0_r, dst1_r, dst2_r, dst3_r; - v8i16 dst10_r, dst32_r, dst21_r, dst43_r; + v8i16 dst30, dst41, dst52, dst63, dst10, dst32, dst54, dst21, dst43, dst65; + v4i32 dst0, dst1, dst2, dst3; src -= (src_stride + 1); @@ -3083,71 +3232,43 @@ static void hevc_hv_4t_4x4_msa(uint8_t *src, SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; const_vec = __msa_ldi_h(128); const_vec <<= 6; - LD_SB3(src, src_stride, src0, src1, src2); - src += (3 * src_stride); - - XORI_B3_128_SB(src0, src1, src2); - - VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); - VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); - VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - - dst0 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); - dst1 = const_vec; - DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); - dst2 = const_vec; - DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); - - ILVR_H2_SH(dst1, dst0, dst2, dst1, dst10_r, dst21_r); - - LD_SB4(src, src_stride, src3, src4, src5, src6); - XORI_B4_128_SB(src3, src4, src5, src6); - - VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); - dst3 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); - - dst32_r = __msa_ilvr_h(dst3, dst2); - dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); - dst0_r >>= 6; - - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - dst43_r = __msa_ilvr_h(dst4, dst3); - dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); - dst1_r >>= 6; - - VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec0, vec1); - dst5 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst5, dst5); - - dst10_r = __msa_ilvr_h(dst5, dst4); - dst2_r = HEVC_FILT_4TAP(dst32_r, dst10_r, filt_h0, filt_h1); - dst2_r >>= 6; - - VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec0, vec1); - dst2 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst2, dst2); + LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); + XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); - dst21_r = __msa_ilvr_h(dst2, dst5); - dst3_r = HEVC_FILT_4TAP(dst43_r, dst21_r, filt_h0, filt_h1); - dst3_r >>= 6; + VSHF_B2_SB(src0, src3, src0, src3, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src1, src4, src1, src4, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src2, src5, src2, src5, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src3, src6, src3, src6, mask0, mask1, vec6, vec7); - PCKEV_H2_SW(dst1_r, dst0_r, dst3_r, dst2_r, dst0_r, dst1_r); - ST8x4_UB(dst0_r, dst1_r, dst, 2 * dst_stride); + dst30 = const_vec; + dst41 = const_vec; + dst52 = const_vec; + dst63 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst30, dst30); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst41, dst41); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst52, dst52); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst63, dst63); + + ILVRL_H2_SH(dst41, dst30, dst10, dst43); + ILVRL_H2_SH(dst52, dst41, dst21, dst54); + ILVRL_H2_SH(dst63, dst52, dst32, dst65); + + dst0 = HEVC_FILT_4TAP(dst10, dst32, filt_h0, filt_h1); + dst1 = HEVC_FILT_4TAP(dst21, dst43, filt_h0, filt_h1); + dst2 = HEVC_FILT_4TAP(dst32, dst54, filt_h0, filt_h1); + dst3 = HEVC_FILT_4TAP(dst43, dst65, filt_h0, filt_h1); + SRA_4V(dst0, dst1, dst2, dst3, 6); + PCKEV_H2_SW(dst1, dst0, dst3, dst2, dst0, dst2); + ST8x4_UB(dst0, dst2, dst, dst_stride_in_bytes); } @@ -3163,25 +3284,24 @@ static void hevc_hv_4t_4multx8mult_msa(uint8_t *src, v16i8 src0, src1, src2, src3, src4, src5, src6; v16i8 src7, src8, src9, src10; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr + 16); v16i8 mask1; v8i16 filter_vec, const_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8, dst9; - v4i32 dst0_r, dst1_r, dst2_r, dst3_r, dst4_r, dst5_r, dst6_r, dst7_r; - v8i16 dst10_r, dst32_r, dst54_r, dst76_r; - v8i16 dst21_r, dst43_r, dst65_r, dst87_r; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v8i16 dst10, dst21, dst22, dst73, dst84, dst95, dst106; + v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst98_r; + v8i16 dst21_r, dst43_r, dst65_r, dst87_r, dst109_r; + v4i32 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; @@ -3191,19 +3311,14 @@ static void hevc_hv_4t_4multx8mult_msa(uint8_t *src, LD_SB3(src, src_stride, src0, src1, src2); src += (3 * src_stride); XORI_B3_128_SB(src0, src1, src2); - - VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); - VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); - VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - - dst0 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); - dst1 = const_vec; - DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); - dst2 = const_vec; - DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); - - ILVR_H2_SH(dst1, dst0, dst2, dst1, dst10_r, dst21_r); + VSHF_B2_SB(src0, src1, src0, src1, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src1, src2, src1, src2, mask0, mask1, vec2, vec3); + dst10 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst10, dst10); + dst21 = const_vec; + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst21, dst21); + ILVRL_H2_SH(dst21, dst10, dst10_r, dst21_r); + dst22 = (v8i16) __msa_splati_d((v2i64) dst21, 1); for (loop_cnt = height >> 3; loop_cnt--;) { LD_SB8(src, src_stride, @@ -3211,75 +3326,45 @@ static void hevc_hv_4t_4multx8mult_msa(uint8_t *src, src += (8 * src_stride); XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); - VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); - dst3 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); - - dst32_r = __msa_ilvr_h(dst3, dst2); - dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); - dst0_r >>= 6; - - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - dst43_r = __msa_ilvr_h(dst4, dst3); - dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); - dst1_r >>= 6; - - VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec0, vec1); - dst5 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst5, dst5); - - dst54_r = __msa_ilvr_h(dst5, dst4); - dst2_r = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); - dst2_r >>= 6; - - VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec0, vec1); - dst6 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst6, dst6); - - dst65_r = __msa_ilvr_h(dst6, dst5); - dst3_r = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); - dst3_r >>= 6; - - VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec0, vec1); - dst7 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst7, dst7); - - dst76_r = __msa_ilvr_h(dst7, dst6); - dst4_r = HEVC_FILT_4TAP(dst54_r, dst76_r, filt_h0, filt_h1); - dst4_r >>= 6; - - VSHF_B2_SB(src8, src8, src8, src8, mask0, mask1, vec0, vec1); - dst8 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst8, dst8); - - dst87_r = __msa_ilvr_h(dst8, dst7); - dst5_r = HEVC_FILT_4TAP(dst65_r, dst87_r, filt_h0, filt_h1); - dst5_r >>= 6; - - VSHF_B2_SB(src9, src9, src9, src9, mask0, mask1, vec0, vec1); - dst9 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst9, dst9); - - dst10_r = __msa_ilvr_h(dst9, dst8); - dst6_r = HEVC_FILT_4TAP(dst76_r, dst10_r, filt_h0, filt_h1); - dst6_r >>= 6; - - VSHF_B2_SB(src10, src10, src10, src10, mask0, mask1, vec0, vec1); - dst2 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst2, dst2); - - dst21_r = __msa_ilvr_h(dst2, dst9); - dst7_r = HEVC_FILT_4TAP(dst87_r, dst21_r, filt_h0, filt_h1); - dst7_r >>= 6; - - PCKEV_H4_SW(dst1_r, dst0_r, dst3_r, dst2_r, - dst5_r, dst4_r, dst7_r, dst6_r, - dst0_r, dst1_r, dst2_r, dst3_r); - ST8x8_UB(dst0_r, dst1_r, dst2_r, dst3_r, dst, 2 * dst_stride); + VSHF_B2_SB(src3, src7, src3, src7, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src4, src8, src4, src8, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src5, src9, src5, src9, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src6, src10, src6, src10, mask0, mask1, vec6, vec7); + + dst73 = const_vec; + dst84 = const_vec; + dst95 = const_vec; + dst106 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst73, dst73); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst84, dst84); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst95, dst95); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst106, dst106); + + dst32_r = __msa_ilvr_h(dst73, dst22); + ILVRL_H2_SH(dst84, dst73, dst43_r, dst87_r); + ILVRL_H2_SH(dst95, dst84, dst54_r, dst98_r); + ILVRL_H2_SH(dst106, dst95, dst65_r, dst109_r); + dst22 = (v8i16) __msa_splati_d((v2i64) dst73, 1); + dst76_r = __msa_ilvr_h(dst22, dst106); + + dst0 = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); + dst1 = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); + dst2 = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); + dst3 = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); + dst4 = HEVC_FILT_4TAP(dst54_r, dst76_r, filt_h0, filt_h1); + dst5 = HEVC_FILT_4TAP(dst65_r, dst87_r, filt_h0, filt_h1); + dst6 = HEVC_FILT_4TAP(dst76_r, dst98_r, filt_h0, filt_h1); + dst7 = HEVC_FILT_4TAP(dst87_r, dst109_r, filt_h0, filt_h1); + SRA_4V(dst0, dst1, dst2, dst3, 6); + SRA_4V(dst4, dst5, dst6, dst7, 6); + PCKEV_H4_SW(dst1, dst0, dst3, dst2, dst5, dst4, dst7, dst6, + dst0, dst1, dst2, dst3); + ST8x8_UB(dst0, dst1, dst2, dst3, dst, 2 * dst_stride); dst += (8 * dst_stride); + + dst10_r = dst98_r; + dst21_r = dst109_r; + dst22 = (v8i16) __msa_splati_d((v2i64) dst106, 1); } } @@ -3311,30 +3396,31 @@ static void hevc_hv_4t_6w_msa(uint8_t *src, const int8_t *filter_y, int32_t height) { - uint32_t loop_cnt; - uint64_t dst_val0, dst_val1, dst_val2, dst_val3; - uint32_t dst_val_int0, dst_val_int1, dst_val_int2, dst_val_int3; - v16i8 src0, src1, src2, src3, src4, src5, src6; + int32_t dst_stride_in_bytes = 2 * dst_stride; + v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v8i16 filter_vec, const_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; - v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; - v8i16 dst10_r, dst32_r, dst21_r, dst43_r; - v8i16 dst10_l, dst32_l, dst21_l, dst43_l; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v8i16 dsth0, dsth1, dsth2, dsth3, dsth4, dsth5, dsth6, dsth7, dsth8, dsth9; + v8i16 dsth10, tmp0, tmp1, tmp2, tmp3, tmp4, tmp5; + v8i16 dst10_r, dst32_r, dst54_r, dst76_r, dst98_r, dst21_r, dst43_r; + v8i16 dst65_r, dst87_r, dst109_r, dst10_l, dst32_l, dst54_l, dst76_l; + v8i16 dst98_l, dst21_l, dst43_l, dst65_l, dst87_l, dst109_l; + v8i16 dst1021_l, dst3243_l, dst5465_l, dst7687_l, dst98109_l; + v4i32 dst0_r, dst1_r, dst2_r, dst3_r, dst4_r, dst5_r, dst6_r, dst7_r; + v4i32 dst0_l, dst1_l, dst2_l, dst3_l; src -= (src_stride + 1); filter_vec = LD_SH(filter_x); SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; @@ -3349,89 +3435,83 @@ static void hevc_hv_4t_6w_msa(uint8_t *src, VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); - dst0 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); - dst1 = const_vec; - DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); - dst2 = const_vec; - DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); - - ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); - ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); - - for (loop_cnt = height >> 2; loop_cnt--;) { - LD_SB4(src, src_stride, src3, src4, src5, src6); - src += (4 * src_stride); - XORI_B4_128_SB(src3, src4, src5, src6); - - VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); - dst3 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); - - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); - dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); - dst0_r >>= 6; - dst0_l >>= 6; - - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); - dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); - dst1_l = HEVC_FILT_4TAP(dst21_l, dst43_l, filt_h0, filt_h1); - dst1_r >>= 6; - dst1_l >>= 6; - - VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec0, vec1); - dst5 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst5, dst5); - - ILVRL_H2_SH(dst5, dst4, dst10_r, dst10_l); - dst2_r = HEVC_FILT_4TAP(dst32_r, dst10_r, filt_h0, filt_h1); - dst2_l = HEVC_FILT_4TAP(dst32_l, dst10_l, filt_h0, filt_h1); - dst2_r >>= 6; - dst2_l >>= 6; - - VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec0, vec1); - dst2 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst2, dst2); - - ILVRL_H2_SH(dst2, dst5, dst21_r, dst21_l); - dst3_r = HEVC_FILT_4TAP(dst43_r, dst21_r, filt_h0, filt_h1); - dst3_l = HEVC_FILT_4TAP(dst43_l, dst21_l, filt_h0, filt_h1); - dst3_r >>= 6; - dst3_l >>= 6; + dsth0 = const_vec; + dsth1 = const_vec; + dsth2 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dsth0, dsth0); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dsth1, dsth1); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dsth2, dsth2); - PCKEV_H4_SW(dst0_l, dst0_r, dst1_l, dst1_r, - dst2_l, dst2_r, dst3_l, dst3_r, - dst0_r, dst1_r, dst2_r, dst3_r); + ILVRL_H2_SH(dsth1, dsth0, dst10_r, dst10_l); + ILVRL_H2_SH(dsth2, dsth1, dst21_r, dst21_l); - dst_val0 = __msa_copy_u_d((v2i64) dst0_r, 0); - dst_val1 = __msa_copy_u_d((v2i64) dst1_r, 0); - dst_val2 = __msa_copy_u_d((v2i64) dst2_r, 0); - dst_val3 = __msa_copy_u_d((v2i64) dst3_r, 0); + LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, src10); + XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); - dst_val_int0 = __msa_copy_u_w((v4i32) dst0_r, 2); - dst_val_int1 = __msa_copy_u_w((v4i32) dst1_r, 2); - dst_val_int2 = __msa_copy_u_w((v4i32) dst2_r, 2); - dst_val_int3 = __msa_copy_u_w((v4i32) dst3_r, 2); + VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); + + dsth3 = const_vec; + dsth4 = const_vec; + dsth5 = const_vec; + dsth6 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dsth3, dsth3); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dsth4, dsth4); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dsth5, dsth5); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dsth6, dsth6); - SD(dst_val0, dst); - SW(dst_val_int0, dst + 4); - dst += dst_stride; - SD(dst_val1, dst); - SW(dst_val_int1, dst + 4); - dst += dst_stride; - SD(dst_val2, dst); - SW(dst_val_int2, dst + 4); - dst += dst_stride; - SD(dst_val3, dst); - SW(dst_val_int3, dst + 4); - dst += dst_stride; + VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src8, src8, src8, src8, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src9, src9, src9, src9, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src10, src10, src10, src10, mask0, mask1, vec6, vec7); + + dsth7 = const_vec; + dsth8 = const_vec; + dsth9 = const_vec; + dsth10 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dsth7, dsth7); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dsth8, dsth8); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dsth9, dsth9); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dsth10, dsth10); + + ILVRL_H2_SH(dsth3, dsth2, dst32_r, dst32_l); + ILVRL_H2_SH(dsth4, dsth3, dst43_r, dst43_l); + ILVRL_H2_SH(dsth5, dsth4, dst54_r, dst54_l); + ILVRL_H2_SH(dsth6, dsth5, dst65_r, dst65_l); + ILVRL_H2_SH(dsth7, dsth6, dst76_r, dst76_l); + ILVRL_H2_SH(dsth8, dsth7, dst87_r, dst87_l); + ILVRL_H2_SH(dsth9, dsth8, dst98_r, dst98_l); + ILVRL_H2_SH(dsth10, dsth9, dst109_r, dst109_l); + + PCKEV_D2_SH(dst21_l, dst10_l, dst43_l, dst32_l, dst1021_l, dst3243_l); + PCKEV_D2_SH(dst65_l, dst54_l, dst87_l, dst76_l, dst5465_l, dst7687_l); + dst98109_l = (v8i16) __msa_pckev_d((v2i64) dst109_l, (v2i64) dst98_l); - } + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); + dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); + dst2_r = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); + dst3_r = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); + dst4_r = HEVC_FILT_4TAP(dst54_r, dst76_r, filt_h0, filt_h1); + dst5_r = HEVC_FILT_4TAP(dst65_r, dst87_r, filt_h0, filt_h1); + dst6_r = HEVC_FILT_4TAP(dst76_r, dst98_r, filt_h0, filt_h1); + dst7_r = HEVC_FILT_4TAP(dst87_r, dst109_r, filt_h0, filt_h1); + dst0_l = HEVC_FILT_4TAP(dst1021_l, dst3243_l, filt_h0, filt_h1); + dst1_l = HEVC_FILT_4TAP(dst3243_l, dst5465_l, filt_h0, filt_h1); + dst2_l = HEVC_FILT_4TAP(dst5465_l, dst7687_l, filt_h0, filt_h1); + dst3_l = HEVC_FILT_4TAP(dst7687_l, dst98109_l, filt_h0, filt_h1); + SRA_4V(dst0_r, dst1_r, dst2_r, dst3_r, 6); + SRA_4V(dst4_r, dst5_r, dst6_r, dst7_r, 6); + SRA_4V(dst0_l, dst1_l, dst2_l, dst3_l, 6); + PCKEV_H2_SH(dst1_r, dst0_r, dst3_r, dst2_r, tmp0, tmp1); + PCKEV_H2_SH(dst5_r, dst4_r, dst7_r, dst6_r, tmp2, tmp3); + PCKEV_H2_SH(dst1_l, dst0_l, dst3_l, dst2_l, tmp4, tmp5); + ST8x4_UB(tmp0, tmp1, dst, dst_stride_in_bytes); + ST4x4_UB(tmp4, tmp4, 0, 1, 2, 3, dst + 4, dst_stride_in_bytes); + dst += 4 * dst_stride; + ST8x4_UB(tmp2, tmp3, dst, dst_stride_in_bytes); + ST4x4_UB(tmp5, tmp5, 0, 1, 2, 3, dst + 4, dst_stride_in_bytes); } static void hevc_hv_4t_8x2_msa(uint8_t *src, @@ -3439,16 +3519,15 @@ static void hevc_hv_4t_8x2_msa(uint8_t *src, int16_t *dst, int32_t dst_stride, const int8_t *filter_x, - const int8_t *filter_y, - int32_t height) + const int8_t *filter_y) { v16i8 src0, src1, src2, src3, src4; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v8i16 filter_vec, const_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9; v8i16 dst0, dst1, dst2, dst3, dst4; v4i32 dst0_r, dst0_l, dst1_r, dst1_l; v8i16 dst10_r, dst32_r, dst21_r, dst43_r; @@ -3460,23 +3539,23 @@ static void hevc_hv_4t_8x2_msa(uint8_t *src, SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; const_vec = __msa_ldi_h(128); const_vec <<= 6; - LD_SB3(src, src_stride, src0, src1, src2); - src += (3 * src_stride); - XORI_B3_128_SB(src0, src1, src2); + LD_SB5(src, src_stride, src0, src1, src2, src3, src4); + XORI_B5_128_SB(src0, src1, src2, src3, src4); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec8, vec9); dst0 = const_vec; DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); @@ -3484,53 +3563,123 @@ static void hevc_hv_4t_8x2_msa(uint8_t *src, DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); dst2 = const_vec; DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); + dst3 = const_vec; + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst3, dst3); + dst4 = const_vec; + DPADD_SB2_SH(vec8, vec9, filt0, filt1, dst4, dst4); ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); - - LD_SB2(src, src_stride, src3, src4); - src += (2 * src_stride); - XORI_B2_128_SB(src3, src4); - - VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); - dst3 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); + ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); - dst0_r >>= 6; - dst0_l >>= 6; - - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); dst1_l = HEVC_FILT_4TAP(dst21_l, dst43_l, filt_h0, filt_h1); - dst1_r >>= 6; - dst1_l >>= 6; - + SRA_4V(dst0_r, dst0_l, dst1_r, dst1_l, 6); PCKEV_H2_SW(dst0_l, dst0_r, dst1_l, dst1_r, dst0_r, dst1_r); ST_SW2(dst0_r, dst1_r, dst, dst_stride); } +static void hevc_hv_4t_8multx4_msa(uint8_t *src, int32_t src_stride, + int16_t *dst, int32_t dst_stride, + const int8_t *filter_x, + const int8_t *filter_y, int32_t width8mult) +{ + int32_t cnt; + v16i8 src0, src1, src2, src3, src4, src5, src6, mask0, mask1; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v8i16 filt0, filt1, filt_h0, filt_h1, filter_vec, const_vec; + v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6; + v8i16 dst10_r, dst32_r, dst54_r, dst21_r, dst43_r, dst65_r; + v8i16 dst10_l, dst32_l, dst54_l, dst21_l, dst43_l, dst65_l; + v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; + + src -= (src_stride + 1); + + filter_vec = LD_SH(filter_x); + SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + + filter_vec = LD_SH(filter_y); + UNPCK_R_SB_SH(filter_vec, filter_vec); + + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); + + mask0 = LD_SB(ff_hevc_mask_arr); + mask1 = mask0 + 2; + + const_vec = __msa_ldi_h(128); + const_vec <<= 6; + + for (cnt = width8mult; cnt--;) { + LD_SB7(src, src_stride, src0, src1, src2, src3, src4, src5, src6); + src += 8; + XORI_B7_128_SB(src0, src1, src2, src3, src4, src5, src6); + + VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); + + dst0 = const_vec; + dst1 = const_vec; + dst2 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); + + ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); + ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); + + VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); + dst3 = const_vec; + dst4 = const_vec; + dst5 = const_vec; + dst6 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst4, dst4); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst5, dst5); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst6, dst6); + ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); + ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); + ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); + ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); + dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); + dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); + dst1_l = HEVC_FILT_4TAP(dst21_l, dst43_l, filt_h0, filt_h1); + + dst2_r = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); + dst2_l = HEVC_FILT_4TAP(dst32_l, dst54_l, filt_h0, filt_h1); + dst3_r = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); + dst3_l = HEVC_FILT_4TAP(dst43_l, dst65_l, filt_h0, filt_h1); + SRA_4V(dst0_r, dst0_l, dst1_r, dst1_l, 6); + SRA_4V(dst2_r, dst2_l, dst3_r, dst3_l, 6); + PCKEV_H2_SW(dst0_l, dst0_r, dst1_l, dst1_r, dst0_r, dst1_r); + PCKEV_H2_SW(dst2_l, dst2_r, dst3_l, dst3_r, dst2_r, dst3_r); + + ST_SW4(dst0_r, dst1_r, dst2_r, dst3_r, dst, dst_stride); + dst += 8; + } +} + static void hevc_hv_4t_8x6_msa(uint8_t *src, int32_t src_stride, int16_t *dst, int32_t dst_stride, const int8_t *filter_x, - const int8_t *filter_y, - int32_t height) + const int8_t *filter_y) { v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v8i16 filter_vec, const_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7, vec8, vec9; + v16i8 vec10, vec11, vec12, vec13, vec14, vec15, vec16, vec17; v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst7, dst8; v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; v4i32 dst4_r, dst4_l, dst5_r, dst5_l; @@ -3545,24 +3694,31 @@ static void hevc_hv_4t_8x6_msa(uint8_t *src, SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; const_vec = __msa_ldi_h(128); const_vec <<= 6; - LD_SB3(src, src_stride, src0, src1, src2); - src += (3 * src_stride); + LD_SB5(src, src_stride, src0, src1, src2, src3, src4); + src += (5 * src_stride); + LD_SB4(src, src_stride, src5, src6, src7, src8); - XORI_B3_128_SB(src0, src1, src2); + XORI_B5_128_SB(src0, src1, src2, src3, src4); + XORI_B4_128_SB(src5, src6, src7, src8); VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec6, vec7); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec8, vec9); + VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec10, vec11); + VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec12, vec13); + VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec14, vec15); + VSHF_B2_SB(src8, src8, src8, src8, mask0, mask1, vec16, vec17); dst0 = const_vec; DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); @@ -3570,91 +3726,44 @@ static void hevc_hv_4t_8x6_msa(uint8_t *src, DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); dst2 = const_vec; DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); + dst3 = const_vec; + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst3, dst3); + dst4 = const_vec; + DPADD_SB2_SH(vec8, vec9, filt0, filt1, dst4, dst4); + dst5 = const_vec; + DPADD_SB2_SH(vec10, vec11, filt0, filt1, dst5, dst5); + dst6 = const_vec; + DPADD_SB2_SH(vec12, vec13, filt0, filt1, dst6, dst6); + dst7 = const_vec; + DPADD_SB2_SH(vec14, vec15, filt0, filt1, dst7, dst7); + dst8 = const_vec; + DPADD_SB2_SH(vec16, vec17, filt0, filt1, dst8, dst8); ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); - - LD_SB2(src, src_stride, src3, src4); - src += (2 * src_stride); - - XORI_B2_128_SB(src3, src4); - - /* row 3 */ - VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); - dst3 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); - ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); - dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); + ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); + ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); + ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); + ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); + ILVRL_H2_SH(dst8, dst7, dst87_r, dst87_l); + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); - - dst0_r >>= 6; - dst0_l >>= 6; - - /* row 4 */ - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); dst1_l = HEVC_FILT_4TAP(dst21_l, dst43_l, filt_h0, filt_h1); - dst1_r >>= 6; - dst1_l >>= 6; - - LD_SB2(src, src_stride, src5, src6); - src += (2 * src_stride); - - XORI_B2_128_SB(src5, src6); - - /* row 5 */ - VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec0, vec1); - dst5 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst5, dst5); - - ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); dst2_r = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); dst2_l = HEVC_FILT_4TAP(dst32_l, dst54_l, filt_h0, filt_h1); - dst2_r >>= 6; - dst2_l >>= 6; - - /* row 6 */ - VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec0, vec1); - dst6 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst6, dst6); - - ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); dst3_r = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); dst3_l = HEVC_FILT_4TAP(dst43_l, dst65_l, filt_h0, filt_h1); - dst3_r >>= 6; - dst3_l >>= 6; - - LD_SB2(src, src_stride, src7, src8); - - XORI_B2_128_SB(src7, src8); - - /* row 7 */ - VSHF_B2_SB(src7, src7, src7, src7, mask0, mask1, vec0, vec1); - dst7 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst7, dst7); - - ILVRL_H2_SH(dst7, dst6, dst76_r, dst76_l); dst4_r = HEVC_FILT_4TAP(dst54_r, dst76_r, filt_h0, filt_h1); dst4_l = HEVC_FILT_4TAP(dst54_l, dst76_l, filt_h0, filt_h1); - dst4_r >>= 6; - dst4_l >>= 6; - - /* row 8 */ - VSHF_B2_SB(src8, src8, src8, src8, mask0, mask1, vec0, vec1); - dst8 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst8, dst8); - - ILVRL_H2_SH(dst8, dst7, dst87_r, dst87_l); dst5_r = HEVC_FILT_4TAP(dst65_r, dst87_r, filt_h0, filt_h1); dst5_l = HEVC_FILT_4TAP(dst65_l, dst87_l, filt_h0, filt_h1); - dst5_r >>= 6; - dst5_l >>= 6; + + SRA_4V(dst0_r, dst0_l, dst1_r, dst1_l, 6); + SRA_4V(dst2_r, dst2_l, dst3_r, dst3_l, 6); + SRA_4V(dst4_r, dst4_l, dst5_r, dst5_l, 6); PCKEV_H4_SW(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, dst3_r, dst0_r, dst1_r, dst2_r, dst3_r); @@ -3674,22 +3783,22 @@ static void hevc_hv_4t_8multx4mult_msa(uint8_t *src, const int8_t *filter_x, const int8_t *filter_y, int32_t height, - int32_t width) + int32_t width8mult) { uint32_t loop_cnt, cnt; uint8_t *src_tmp; int16_t *dst_tmp; v16i8 src0, src1, src2, src3, src4, src5, src6; v8i16 filt0, filt1; - v4i32 filt_h0, filt_h1; - v16i8 mask0 = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8 }; + v8i16 filt_h0, filt_h1; + v16i8 mask0 = LD_SB(ff_hevc_mask_arr); v16i8 mask1; v8i16 filter_vec, const_vec; - v16i8 vec0, vec1, vec2, vec3, vec4, vec5; - v8i16 dst0, dst1, dst2, dst3, dst4, dst5; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6; v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; - v8i16 dst10_r, dst32_r, dst21_r, dst43_r; - v8i16 dst10_l, dst32_l, dst21_l, dst43_l; + v8i16 dst10_r, dst32_r, dst54_r, dst21_r, dst43_r, dst65_r; + v8i16 dst10_l, dst32_l, dst54_l, dst21_l, dst43_l, dst65_l; src -= (src_stride + 1); @@ -3697,17 +3806,16 @@ static void hevc_hv_4t_8multx4mult_msa(uint8_t *src, SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); filter_vec = LD_SH(filter_y); - vec0 = __msa_clti_s_b((v16i8) filter_vec, 0); - filter_vec = (v8i16) __msa_ilvr_b(vec0, (v16i8) filter_vec); + UNPCK_R_SB_SH(filter_vec, filter_vec); - SPLATI_W2_SW(filter_vec, 0, filt_h0, filt_h1); + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); mask1 = mask0 + 2; const_vec = __msa_ldi_h(128); const_vec <<= 6; - for (cnt = width >> 3; cnt--;) { + for (cnt = width8mult; cnt--;) { src_tmp = src; dst_tmp = dst; @@ -3736,59 +3844,48 @@ static void hevc_hv_4t_8multx4mult_msa(uint8_t *src, XORI_B4_128_SB(src3, src4, src5, src6); VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); + dst3 = const_vec; + dst4 = const_vec; + dst5 = const_vec; + dst6 = const_vec; DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst4, dst4); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst5, dst5); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst6, dst6); ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); + ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); + ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); + ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); - - dst0_r >>= 6; - dst0_l >>= 6; - - /* row 4 */ - VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec0, vec1); - dst4 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst4, dst4); - - ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); dst1_l = HEVC_FILT_4TAP(dst21_l, dst43_l, filt_h0, filt_h1); - dst1_r >>= 6; - dst1_l >>= 6; - - /* row 5 */ - VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec0, vec1); - dst5 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst5, dst5); + dst2_r = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); + dst2_l = HEVC_FILT_4TAP(dst32_l, dst54_l, filt_h0, filt_h1); + dst3_r = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); + dst3_l = HEVC_FILT_4TAP(dst43_l, dst65_l, filt_h0, filt_h1); - ILVRL_H2_SH(dst5, dst4, dst10_r, dst10_l); - dst2_r = HEVC_FILT_4TAP(dst32_r, dst10_r, filt_h0, filt_h1); - dst2_l = HEVC_FILT_4TAP(dst32_l, dst10_l, filt_h0, filt_h1); - - dst2_r >>= 6; - dst2_l >>= 6; - - /* row 6 */ - VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec0, vec1); - dst2 = const_vec; - DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst2, dst2); - - ILVRL_H2_SH(dst2, dst5, dst21_r, dst21_l); - dst3_r = HEVC_FILT_4TAP(dst43_r, dst21_r, filt_h0, filt_h1); - dst3_l = HEVC_FILT_4TAP(dst43_l, dst21_l, filt_h0, filt_h1); - - dst3_r >>= 6; - dst3_l >>= 6; + SRA_4V(dst0_r, dst0_l, dst1_r, dst1_l, 6); + SRA_4V(dst2_r, dst2_l, dst3_r, dst3_l, 6); PCKEV_H4_SW(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, dst3_r, dst0_r, dst1_r, dst2_r, dst3_r); - ST_SW2(dst0_r, dst1_r, dst_tmp, dst_stride); - dst_tmp += (2 * dst_stride); - ST_SW2(dst2_r, dst3_r, dst_tmp, dst_stride); - dst_tmp += (2 * dst_stride); + ST_SW4(dst0_r, dst1_r, dst2_r, dst3_r, dst_tmp, dst_stride); + dst_tmp += (4 * dst_stride); + + dst10_r = dst54_r; + dst10_l = dst54_l; + dst21_r = dst65_r; + dst21_l = dst65_l; + dst2 = dst6; } src += 8; @@ -3807,13 +3904,16 @@ static void hevc_hv_4t_8w_msa(uint8_t *src, if (2 == height) { hevc_hv_4t_8x2_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height); + filter_x, filter_y); + } else if (4 == height) { + hevc_hv_4t_8multx4_msa(src, src_stride, dst, dst_stride, + filter_x, filter_y, 1); } else if (6 == height) { hevc_hv_4t_8x6_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height); + filter_x, filter_y); } else if (0 == (height % 4)) { hevc_hv_4t_8multx4mult_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height, 8); + filter_x, filter_y, height, 1); } } @@ -3825,12 +3925,169 @@ static void hevc_hv_4t_12w_msa(uint8_t *src, const int8_t *filter_y, int32_t height) { - hevc_hv_4t_8multx4mult_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height, 8); + uint32_t loop_cnt; + uint8_t *src_tmp; + int16_t *dst_tmp; + v16i8 src0, src1, src2, src3, src4, src5, src6, src7, src8, src9, src10; + v16i8 vec0, vec1, vec2, vec3, vec4, vec5, vec6, vec7; + v16i8 mask0, mask1, mask2, mask3; + v8i16 filt0, filt1, filt_h0, filt_h1, filter_vec, const_vec; + v8i16 dst0, dst1, dst2, dst3, dst4, dst5, dst6, dst10, dst21, dst22, dst73; + v8i16 dst84, dst95, dst106, dst76_r, dst98_r, dst87_r, dst109_r; + v8i16 dst10_r, dst32_r, dst54_r, dst21_r, dst43_r, dst65_r; + v8i16 dst10_l, dst32_l, dst54_l, dst21_l, dst43_l, dst65_l; + v4i32 dst0_r, dst0_l, dst1_r, dst1_l, dst2_r, dst2_l, dst3_r, dst3_l; + v4i32 tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7; + + src -= (src_stride + 1); + + filter_vec = LD_SH(filter_x); + SPLATI_H2_SH(filter_vec, 0, 1, filt0, filt1); + + filter_vec = LD_SH(filter_y); + UNPCK_R_SB_SH(filter_vec, filter_vec); + + SPLATI_W2_SH(filter_vec, 0, filt_h0, filt_h1); + + mask0 = LD_SB(ff_hevc_mask_arr); + mask1 = mask0 + 2; + + const_vec = __msa_ldi_h(128); + const_vec <<= 6; + + src_tmp = src; + dst_tmp = dst; + + LD_SB3(src_tmp, src_stride, src0, src1, src2); + src_tmp += (3 * src_stride); + + XORI_B3_128_SB(src0, src1, src2); + + VSHF_B2_SB(src0, src0, src0, src0, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src1, src1, src1, src1, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src2, src2, src2, src2, mask0, mask1, vec4, vec5); + + dst0 = const_vec; + dst1 = const_vec; + dst2 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst0, dst0); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst1, dst1); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst2, dst2); - hevc_hv_4t_4w_msa(src + 8, src_stride, dst + 8, dst_stride, - filter_x, filter_y, height); + ILVRL_H2_SH(dst1, dst0, dst10_r, dst10_l); + ILVRL_H2_SH(dst2, dst1, dst21_r, dst21_l); + + for (loop_cnt = 4; loop_cnt--;) { + LD_SB4(src_tmp, src_stride, src3, src4, src5, src6); + src_tmp += (4 * src_stride); + XORI_B4_128_SB(src3, src4, src5, src6); + + VSHF_B2_SB(src3, src3, src3, src3, mask0, mask1, vec0, vec1); + VSHF_B2_SB(src4, src4, src4, src4, mask0, mask1, vec2, vec3); + VSHF_B2_SB(src5, src5, src5, src5, mask0, mask1, vec4, vec5); + VSHF_B2_SB(src6, src6, src6, src6, mask0, mask1, vec6, vec7); + dst3 = const_vec; + dst4 = const_vec; + dst5 = const_vec; + dst6 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst3, dst3); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst4, dst4); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst5, dst5); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst6, dst6); + + ILVRL_H2_SH(dst3, dst2, dst32_r, dst32_l); + ILVRL_H2_SH(dst4, dst3, dst43_r, dst43_l); + ILVRL_H2_SH(dst5, dst4, dst54_r, dst54_l); + ILVRL_H2_SH(dst6, dst5, dst65_r, dst65_l); + + dst0_r = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); + dst0_l = HEVC_FILT_4TAP(dst10_l, dst32_l, filt_h0, filt_h1); + dst1_r = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); + dst1_l = HEVC_FILT_4TAP(dst21_l, dst43_l, filt_h0, filt_h1); + dst2_r = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); + dst2_l = HEVC_FILT_4TAP(dst32_l, dst54_l, filt_h0, filt_h1); + dst3_r = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); + dst3_l = HEVC_FILT_4TAP(dst43_l, dst65_l, filt_h0, filt_h1); + + SRA_4V(dst0_r, dst0_l, dst1_r, dst1_l, 6); + SRA_4V(dst2_r, dst2_l, dst3_r, dst3_l, 6); + PCKEV_H4_SW(dst0_l, dst0_r, dst1_l, dst1_r, dst2_l, dst2_r, dst3_l, + dst3_r, dst0_r, dst1_r, dst2_r, dst3_r); + ST_SW4(dst0_r, dst1_r, dst2_r, dst3_r, dst_tmp, dst_stride); + dst_tmp += (4 * dst_stride); + + dst10_r = dst54_r; + dst10_l = dst54_l; + dst21_r = dst65_r; + dst21_l = dst65_l; + dst2 = dst6; + } + + src += 8; + dst += 8; + + mask2 = LD_SB(ff_hevc_mask_arr + 16); + mask3 = mask2 + 2; + + LD_SB3(src, src_stride, src0, src1, src2); + src += (3 * src_stride); + XORI_B3_128_SB(src0, src1, src2); + VSHF_B2_SB(src0, src1, src0, src1, mask2, mask3, vec0, vec1); + VSHF_B2_SB(src1, src2, src1, src2, mask2, mask3, vec2, vec3); + dst10 = const_vec; + dst21 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst10, dst10); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst21, dst21); + ILVRL_H2_SH(dst21, dst10, dst10_r, dst21_r); + dst22 = (v8i16) __msa_splati_d((v2i64) dst21, 1); + + for (loop_cnt = 2; loop_cnt--;) { + LD_SB8(src, src_stride, src3, src4, src5, src6, src7, src8, src9, + src10); + src += (8 * src_stride); + XORI_B8_128_SB(src3, src4, src5, src6, src7, src8, src9, src10); + VSHF_B2_SB(src3, src7, src3, src7, mask2, mask3, vec0, vec1); + VSHF_B2_SB(src4, src8, src4, src8, mask2, mask3, vec2, vec3); + VSHF_B2_SB(src5, src9, src5, src9, mask2, mask3, vec4, vec5); + VSHF_B2_SB(src6, src10, src6, src10, mask2, mask3, vec6, vec7); + + dst73 = const_vec; + dst84 = const_vec; + dst95 = const_vec; + dst106 = const_vec; + DPADD_SB2_SH(vec0, vec1, filt0, filt1, dst73, dst73); + DPADD_SB2_SH(vec2, vec3, filt0, filt1, dst84, dst84); + DPADD_SB2_SH(vec4, vec5, filt0, filt1, dst95, dst95); + DPADD_SB2_SH(vec6, vec7, filt0, filt1, dst106, dst106); + + dst32_r = __msa_ilvr_h(dst73, dst22); + ILVRL_H2_SH(dst84, dst73, dst43_r, dst87_r); + ILVRL_H2_SH(dst95, dst84, dst54_r, dst98_r); + ILVRL_H2_SH(dst106, dst95, dst65_r, dst109_r); + dst22 = (v8i16) __msa_splati_d((v2i64) dst73, 1); + dst76_r = __msa_ilvr_h(dst22, dst106); + + tmp0 = HEVC_FILT_4TAP(dst10_r, dst32_r, filt_h0, filt_h1); + tmp1 = HEVC_FILT_4TAP(dst21_r, dst43_r, filt_h0, filt_h1); + tmp2 = HEVC_FILT_4TAP(dst32_r, dst54_r, filt_h0, filt_h1); + tmp3 = HEVC_FILT_4TAP(dst43_r, dst65_r, filt_h0, filt_h1); + tmp4 = HEVC_FILT_4TAP(dst54_r, dst76_r, filt_h0, filt_h1); + tmp5 = HEVC_FILT_4TAP(dst65_r, dst87_r, filt_h0, filt_h1); + tmp6 = HEVC_FILT_4TAP(dst76_r, dst98_r, filt_h0, filt_h1); + tmp7 = HEVC_FILT_4TAP(dst87_r, dst109_r, filt_h0, filt_h1); + + SRA_4V(tmp0, tmp1, tmp2, tmp3, 6); + SRA_4V(tmp4, tmp5, tmp6, tmp7, 6); + PCKEV_H4_SW(tmp1, tmp0, tmp3, tmp2, tmp5, tmp4, tmp7, tmp6, tmp0, tmp1, + tmp2, tmp3); + ST8x8_UB(tmp0, tmp1, tmp2, tmp3, dst, 2 * dst_stride); + dst += (8 * dst_stride); + + dst10_r = dst98_r; + dst21_r = dst109_r; + dst22 = (v8i16) __msa_splati_d((v2i64) dst106, 1); + } } static void hevc_hv_4t_16w_msa(uint8_t *src, @@ -3841,8 +4098,13 @@ static void hevc_hv_4t_16w_msa(uint8_t *src, const int8_t *filter_y, int32_t height) { - hevc_hv_4t_8multx4mult_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height, 16); + if (4 == height) { + hevc_hv_4t_8multx4_msa(src, src_stride, dst, dst_stride, + filter_x, filter_y, 2); + } else { + hevc_hv_4t_8multx4mult_msa(src, src_stride, dst, dst_stride, + filter_x, filter_y, height, 2); + } } static void hevc_hv_4t_24w_msa(uint8_t *src, @@ -3854,7 +4116,7 @@ static void hevc_hv_4t_24w_msa(uint8_t *src, int32_t height) { hevc_hv_4t_8multx4mult_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height, 24); + filter_x, filter_y, height, 3); } static void hevc_hv_4t_32w_msa(uint8_t *src, @@ -3866,7 +4128,7 @@ static void hevc_hv_4t_32w_msa(uint8_t *src, int32_t height) { hevc_hv_4t_8multx4mult_msa(src, src_stride, dst, dst_stride, - filter_x, filter_y, height, 32); + filter_x, filter_y, height, 4); } #define MC_COPY(WIDTH) \ @@ -3893,19 +4155,19 @@ MC_COPY(64); #undef MC_COPY -#define MC(PEL, DIR, WIDTH, TAP, DIR1, FILT_DIR) \ -void ff_hevc_put_hevc_##PEL##_##DIR##WIDTH##_8_msa(int16_t *dst, \ - uint8_t *src, \ - ptrdiff_t src_stride, \ - int height, \ - intptr_t mx, \ - intptr_t my, \ - int width) \ -{ \ - const int8_t *filter = ff_hevc_##PEL##_filters[FILT_DIR - 1]; \ - \ - hevc_##DIR1##_##TAP##t_##WIDTH##w_msa(src, src_stride, dst, \ - MAX_PB_SIZE, filter, height); \ +#define MC(PEL, DIR, WIDTH, TAP, DIR1, FILT_DIR) \ +void ff_hevc_put_hevc_##PEL##_##DIR##WIDTH##_8_msa(int16_t *dst, \ + uint8_t *src, \ + ptrdiff_t src_stride, \ + int height, \ + intptr_t mx, \ + intptr_t my, \ + int width) \ +{ \ + const int8_t *filter = ff_hevc_##PEL##_filters[FILT_DIR - 1]; \ + \ + hevc_##DIR1##_##TAP##t_##WIDTH##w_msa(src, src_stride, dst, \ + MAX_PB_SIZE, filter, height); \ } MC(qpel, h, 4, 8, hz, mx); @@ -3944,37 +4206,37 @@ MC(epel, v, 32, 4, vt, my); #undef MC -#define MC_HV(PEL, DIR, WIDTH, TAP, DIR1) \ -void ff_hevc_put_hevc_##PEL##_##DIR##WIDTH##_8_msa(int16_t *dst, \ - uint8_t *src, \ - ptrdiff_t src_stride, \ - int height, \ - intptr_t mx, \ - intptr_t my, \ - int width) \ -{ \ - const int8_t *filter_x = ff_hevc_##PEL##_filters[mx - 1]; \ - const int8_t *filter_y = ff_hevc_##PEL##_filters[my - 1]; \ - \ - hevc_##DIR1##_##TAP##t_##WIDTH##w_msa(src, src_stride, dst, MAX_PB_SIZE, \ - filter_x, filter_y, height); \ -} - -MC_HV(qpel, hv, 4, 8, hv); -MC_HV(qpel, hv, 8, 8, hv); -MC_HV(qpel, hv, 12, 8, hv); -MC_HV(qpel, hv, 16, 8, hv); -MC_HV(qpel, hv, 24, 8, hv); -MC_HV(qpel, hv, 32, 8, hv); -MC_HV(qpel, hv, 48, 8, hv); -MC_HV(qpel, hv, 64, 8, hv); - -MC_HV(epel, hv, 4, 4, hv); -MC_HV(epel, hv, 6, 4, hv); -MC_HV(epel, hv, 8, 4, hv); -MC_HV(epel, hv, 12, 4, hv); -MC_HV(epel, hv, 16, 4, hv); -MC_HV(epel, hv, 24, 4, hv); -MC_HV(epel, hv, 32, 4, hv); +#define MC_HV(PEL, WIDTH, TAP) \ +void ff_hevc_put_hevc_##PEL##_hv##WIDTH##_8_msa(int16_t *dst, \ + uint8_t *src, \ + ptrdiff_t src_stride, \ + int height, \ + intptr_t mx, \ + intptr_t my, \ + int width) \ +{ \ + const int8_t *filter_x = ff_hevc_##PEL##_filters[mx - 1]; \ + const int8_t *filter_y = ff_hevc_##PEL##_filters[my - 1]; \ + \ + hevc_hv_##TAP##t_##WIDTH##w_msa(src, src_stride, dst, MAX_PB_SIZE, \ + filter_x, filter_y, height); \ +} + +MC_HV(qpel, 4, 8); +MC_HV(qpel, 8, 8); +MC_HV(qpel, 12, 8); +MC_HV(qpel, 16, 8); +MC_HV(qpel, 24, 8); +MC_HV(qpel, 32, 8); +MC_HV(qpel, 48, 8); +MC_HV(qpel, 64, 8); + +MC_HV(epel, 4, 4); +MC_HV(epel, 6, 4); +MC_HV(epel, 8, 4); +MC_HV(epel, 12, 4); +MC_HV(epel, 16, 4); +MC_HV(epel, 24, 4); +MC_HV(epel, 32, 4); #undef MC_HV