From patchwork Sun Dec 4 21:34:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philip Langdale X-Patchwork-Id: 39587 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp2812377pzb; Sun, 4 Dec 2022 13:36:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf4CaNhlCV+KLf58Vag0UmyZQZzqd7wyBd93K6LzyOcAMe2ROhx4kDiL79upMOfg8bEgITVg X-Received: by 2002:a17:907:d08c:b0:7b6:62c:dd57 with SMTP id vc12-20020a170907d08c00b007b6062cdd57mr4332315ejc.207.1670189763812; Sun, 04 Dec 2022 13:36:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670189763; cv=none; d=google.com; s=arc-20160816; b=UOIkqoYA9+e0b2j4HJRnuFbXI903FaPZ8WKrwKQ4GyQ+SrKlU5iXIBWWaj+xrW9lxG HcOgFPjadM7O64X6TGEVeK/L8G+pG3Rnzkjy1A7amj4mF6ONC1Yjn8eFxfzbiIzuGkFb XpBLgMv2YehVFvgAOrD9w0Trtuke1MIZPzicfu5AnO5mtoZe49Y9W63A8YoObvnfShi9 7zRfM7WLHL+0P51wF1FKjh5z3g3mxeeWMY1nA7ucHlYIPMheJ7zFdq1EVUTKrOMcUZ/W vrL5X9b/aq4Z9PUbW4Dp91tvcu51PNmLeIp1bX2vbuAM9LjGj3z0OYqmdO8TuD7FRrOg idCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=XPE3i5zg8XtHQAjgbSuz3liSAWDfix+za4vo/ot0Z+o=; b=HmMCe+cVInHj66C/Mle/erDYdBefit/RqEATSKGEifIRR1Mi5ZGL4G1I3FR6WUqk4Z ZwJnU/PfjUBqiWWrvHxwtIN2Kd1WBsDWX9Fs/CW92pKkxhBftzjz7fuUPVRZdbrGp8kH MtTZVUsz1szwvASTm5J4dBdEFY96sulWZf3p73d3F2qy6ZViTCcWw4NBLeaa0oUiUzUV NwNpniL+dNVAZ8QfUaZz1fMe1LxwiBhcB0f4t/kfhsYFuXLGlkhyV4fKFrai5i3tvZil NQgus9Yq//WFs08i69M3oXL576TY+PrQNtR3RQLhM2twFRQMueFM30WgEWFIHdn547Y5 lqUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@overt.org header.s=mail header.b=FifjWUtt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=overt.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t19-20020a1709067c1300b007ad9885d2b1si8219093ejo.301.2022.12.04.13.36.01; Sun, 04 Dec 2022 13:36:03 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@overt.org header.s=mail header.b=FifjWUtt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=overt.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 82748688288; Sun, 4 Dec 2022 23:35:58 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.overt.org (mail.overt.org [72.14.183.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 71E07688288 for ; Sun, 4 Dec 2022 23:35:51 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=overt.org; s=mail; t=1670189748; bh=izPAUIvn3Z8bwv6b8xpaaa0135Bz4sOLT4jKU5yV9rs=; h=From:To:Cc:Subject:Date:From; b=FifjWUttGtdjPAj94IKr3DG4ODbMYYJxMCMgcRns2JfCZk2bHWyvpwbNXZ5mpyJ5u wRwt9+wg4dbouCM5mKrqnP/4kr0VmahiTDr7WphKKWNfzTn6OK6dE6VMBMTUnDlrFJ Hib3xlrscEIO0Mux6OFieoXuWlkeT1Vixfo77xwBgcGoK3uN6Ryun6Z0wTqOjgQagI rtgCrEumnDpKkZ+8zyjXsc53W394eHGRiuk9uVc/OPLdv9LXYeecUvjE9FLwV7jdH7 DXV31mXfTfvCBTMddNVk71qR4Z2qPvc1nHtdeVl7usPv0gWAgPD3heoc5UIsJBKNu1 kt0U7yfRe1Q+g== Received: from authenticated-user (mail.overt.org [72.14.183.176]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.overt.org (Postfix) with ESMTPSA id 8B534629F5; Sun, 4 Dec 2022 15:35:48 -0600 (CST) From: Philip Langdale To: ffmpeg-devel@ffmpeg.org Date: Sun, 4 Dec 2022 13:34:58 -0800 Message-Id: <20221204213458.79978-1-philipl@overt.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] lavu/pixdesc: handle xv30be in av_[read|write]_image_line X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Philip Langdale Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: EKXA0aVts5j4 xv30be is an obnoxious format that I shouldn't have included in the first place. xv30 packs 3 10bit channels into 32bits and while our byte-oriented logic can handle Little Endian correctly, it cannot handle Big Endian. To avoid that, I marked xv30be as a bitstream format, but while that didn't produce FATE errors, it turns out that the existing read/write code silently produces incorrect results, which can be revealed via ubsan. In all likelyhood, the correct fix here is to remove the format. As this format is only used by Intel vaapi, it's only going to show up in LE form, so we could just drop the BE version. But I don't want to deal with creating a hole in the pixfmt list and all the weirdness that comes from that. Instead, I decided to write the correct read/write code for it. And that code isn't too bad, as long as it's specialised for this format, as the channels are all bit-aligned inside a 32bit word. Signed-off-by: Philip Langdale --- libavutil/pixdesc.c | 70 ++++++++++++++++++++++++++++++++------------- 1 file changed, 50 insertions(+), 20 deletions(-) diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c index ca3e204a0b..62a2ae08d9 100644 --- a/libavutil/pixdesc.c +++ b/libavutil/pixdesc.c @@ -46,19 +46,35 @@ void av_read_image_line2(void *dst, uint32_t *dst32 = dst; if (flags & AV_PIX_FMT_FLAG_BITSTREAM) { - int skip = x * step + comp.offset; - const uint8_t *p = data[plane] + y * linesize[plane] + (skip >> 3); - int shift = 8 - depth - (skip & 7); + if (depth == 10) { + // Assume all channels are packed into a 32bit value + const uint8_t *byte_p = data[plane] + y * linesize[plane]; + const uint32_t *p = (uint32_t *)byte_p; - while (w--) { - int val = (*p >> shift) & mask; - if (read_pal_component) - val = data[1][4*val + c]; - shift -= step; - p -= shift >> 3; - shift &= 7; - if (dst_element_size == 4) *dst32++ = val; - else *dst16++ = val; + while (w--) { + int val = AV_RB32(p); + val = (val >> comp.offset) & mask; + if (read_pal_component) + val = data[1][4*val + c]; + if (dst_element_size == 4) *dst32++ = val; + else *dst16++ = val; + p++; + } + } else { + int skip = x * step + comp.offset; + const uint8_t *p = data[plane] + y * linesize[plane] + (skip >> 3); + int shift = 8 - depth - (skip & 7); + + while (w--) { + int val = (*p >> shift) & mask; + if (read_pal_component) + val = data[1][4*val + c]; + shift -= step; + p -= shift >> 3; + shift &= 7; + if (dst_element_size == 4) *dst32++ = val; + else *dst16++ = val; + } } } else { const uint8_t *p = data[plane] + y * linesize[plane] + @@ -109,15 +125,29 @@ void av_write_image_line2(const void *src, const uint16_t *src16 = src; if (flags & AV_PIX_FMT_FLAG_BITSTREAM) { - int skip = x * step + comp.offset; - uint8_t *p = data[plane] + y * linesize[plane] + (skip >> 3); - int shift = 8 - depth - (skip & 7); + if (depth == 10) { + // Assume all channels are packed into a 32bit value + const uint8_t *byte_p = data[plane] + y * linesize[plane]; + uint32_t *p = (uint32_t *)byte_p; + int offset = comp.offset; + uint32_t mask = ((1ULL << depth) - 1) << offset; - while (w--) { - *p |= (src_element_size == 4 ? *src32++ : *src16++) << shift; - shift -= step; - p -= shift >> 3; - shift &= 7; + while (w--) { + uint16_t val = src_element_size == 4 ? *src32++ : *src16++; + AV_WB32(p, (AV_RB32(p) & ~mask) | (val << offset)); + p++; + } + } else { + int skip = x * step + comp.offset; + uint8_t *p = data[plane] + y * linesize[plane] + (skip >> 3); + int shift = 8 - depth - (skip & 7); + + while (w--) { + *p |= (src_element_size == 4 ? *src32++ : *src16++) << shift; + shift -= step; + p -= shift >> 3; + shift &= 7; + } } } else { int shift = comp.shift;