From patchwork Thu Aug 3 15:21:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Cowgill X-Patchwork-Id: 4615 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.46.211 with SMTP id u202csp1029798vsu; Thu, 3 Aug 2017 08:22:15 -0700 (PDT) X-Received: by 10.223.147.97 with SMTP id 88mr1640251wro.226.1501773735662; Thu, 03 Aug 2017 08:22:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1501773735; cv=none; d=google.com; s=arc-20160816; b=DX+okX/t/nFsTKC8zzagn1oovOVlAXMtVbSa08mkUKu5d4efnzSvuVbYigkPsXz1xI 7otU9cgxfoJ//muA5ndn1esyDcmR6OmxDgbCoU9qGyhtZsiMP0FS4OyUGm4o9oX6iut7 YTi3XjFIb4QfS5FfdxSbNB0BttNOpI688ZAzCBt/kReAEOJKlLZ0jtCbvdgbqTGCo9a5 Oj3fqbyd9nA+EMMvdjWvsXzlmv7lsdBunrOXgf4maujIiy6ED7qnElRyf8OljxKt1MDT J+RN2sL4XO1ZoF1Y/CmpRrOMTMyFgFdwW4qCjKVILGHnuxWaffYd/4DWJrr6CO6Lq2NF LHcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to:arc-authentication-results; bh=s+roaVSnNxAePKZBTKQx7QsVd9kPZdmqNUjLmbAiCBk=; b=0wzq3ff1UnwPt0u1lbta03q3vOarkRL3MkM+4j/tR7wtBDbhkZ1w6TvuvFnvSyAxLK DsTuyhwY4OKwkAlXeL/R2aPBZnyDOj8OiwUHrfi31FuWzlCeu3mRSKwtd3hTiBU6x7xH QJ6CaWAq8HBmuEJBbFMSwo9ITaSgxmfzXDONBpfYYmXSK+4IsC3ep2LXFBEu2QYN+b8w fgWmwPd7VM+KoZvvJMDnNC67GF+7Ws477NcmZ8Lquu0MkOxRIL+zXs8HPCwL2lv5ia6o 26wN/gAvSbZ6g6ca9BwzPFUBfcvBZ7LEljgsBC6NAAKgyp1WwK5fo/SDIJxbJ6R1CO4C 2PgQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y17si1453997wmd.242.2017.08.03.08.22.14; Thu, 03 Aug 2017 08:22:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1482C68048C; Thu, 3 Aug 2017 18:22:08 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mailapp01.imgtec.com (mailapp01.imgtec.com [195.59.15.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C2BC268048C for ; Thu, 3 Aug 2017 18:22:01 +0300 (EEST) Received: from HHMAIL01.hh.imgtec.org (unknown [10.100.10.19]) by Forcepoint Email with ESMTPS id C6B7B67138DF6 for ; Thu, 3 Aug 2017 16:22:00 +0100 (IST) Received: from LDT-J-COWGILL.le.imgtec.org (10.150.130.85) by HHMAIL01.hh.imgtec.org (10.100.10.21) with Microsoft SMTP Server (TLS) id 14.3.294.0; Thu, 3 Aug 2017 16:22:04 +0100 From: James Cowgill To: Date: Thu, 3 Aug 2017 16:21:54 +0100 Message-ID: <20170803152154.28024-1-James.Cowgill@imgtec.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <098d4d27-8082-7f9a-40c7-4a9608667117@imgtec.com> References: <098d4d27-8082-7f9a-40c7-4a9608667117@imgtec.com> MIME-Version: 1.0 X-Originating-IP: [10.150.130.85] Subject: [FFmpeg-devel] [PATCH v2] swscale: fix gbrap16 alpha channel issues X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes filter-pixfmts-scale test failing on big-endian systems due to alpSrc not being cast to (const int32_t**). Also fixes distortions in the output alpha channel values by copying the alpha channel code from the rgba64 case found elsewhere in output.c. Fixes ticket 6555. Signed-off-by: James Cowgill --- v2 ---- Move declaration of A inside the loop and don't bother initializing it since the initial value would never be read. libswscale/output.c | 16 ++++++++-------- tests/ref/fate/filter-pixfmts-scale | 4 ++-- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/libswscale/output.c b/libswscale/output.c index 9774e9f327..f30bce8dd3 100644 --- a/libswscale/output.c +++ b/libswscale/output.c @@ -2026,24 +2026,24 @@ yuv2gbrp16_full_X_c(SwsContext *c, const int16_t *lumFilter, const int16_t **lumSrcx, int lumFilterSize, const int16_t *chrFilter, const int16_t **chrUSrcx, const int16_t **chrVSrcx, int chrFilterSize, - const int16_t **alpSrc, uint8_t **dest, + const int16_t **alpSrcx, uint8_t **dest, int dstW, int y) { const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(c->dstFormat); int i; - int hasAlpha = (desc->flags & AV_PIX_FMT_FLAG_ALPHA) && alpSrc; + int hasAlpha = (desc->flags & AV_PIX_FMT_FLAG_ALPHA) && alpSrcx; uint16_t **dest16 = (uint16_t**)dest; const int32_t **lumSrc = (const int32_t**)lumSrcx; const int32_t **chrUSrc = (const int32_t**)chrUSrcx; const int32_t **chrVSrc = (const int32_t**)chrVSrcx; - int A = 0; // init to silence warning + const int32_t **alpSrc = (const int32_t**)alpSrcx; for (i = 0; i < dstW; i++) { int j; int Y = -0x40000000; int U = -(128 << 23); int V = -(128 << 23); - int R, G, B; + int R, G, B, A; for (j = 0; j < lumFilterSize; j++) Y += lumSrc[j][i] * (unsigned)lumFilter[j]; @@ -2059,13 +2059,13 @@ yuv2gbrp16_full_X_c(SwsContext *c, const int16_t *lumFilter, V >>= 14; if (hasAlpha) { - A = 1 << 18; + A = -0x40000000; for (j = 0; j < lumFilterSize; j++) A += alpSrc[j][i] * lumFilter[j]; - if (A & 0xF8000000) - A = av_clip_uintp2(A, 27); + A >>= 1; + A += 0x20002000; } Y -= c->yuv2rgb_y_offset; @@ -2083,7 +2083,7 @@ yuv2gbrp16_full_X_c(SwsContext *c, const int16_t *lumFilter, dest16[1][i] = B >> 14; dest16[2][i] = R >> 14; if (hasAlpha) - dest16[3][i] = A >> 11; + dest16[3][i] = av_clip_uintp2(A, 30) >> 14; } if ((!isBE(c->dstFormat)) != (!HAVE_BIGENDIAN)) { for (i = 0; i < dstW; i++) { diff --git a/tests/ref/fate/filter-pixfmts-scale b/tests/ref/fate/filter-pixfmts-scale index 9b601b71da..dcc34bd4d1 100644 --- a/tests/ref/fate/filter-pixfmts-scale +++ b/tests/ref/fate/filter-pixfmts-scale @@ -23,8 +23,8 @@ gbrap10be 6d89abb9248006c3e9017545e9474654 gbrap10le cf974e23f485a10740f5de74a5c8c3df gbrap12be 1d9b57766ba9c2192403f43967cb9af0 gbrap12le bb1ba1c157717db3dd612a76d38a018e -gbrap16be 81542b96575d1fe3b239d23899f5ece3 -gbrap16le 6feb8b9da131917abe867e0eaaf07b90 +gbrap16be c72b935a6e57a8e1c37bff08c2db55b1 +gbrap16le 13eb0e62b1ac9c1c86c81521eaefab5f gbrp dc3387f925f972c61aae7eb23cdc19f0 gbrp10be 0277d4c3a8498d75e2783fb81379e481 gbrp10le f3d70f8ab845c3c9b8f7452e4a6e285a