From patchwork Mon Sep 26 16:11:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chema Gonzalez X-Patchwork-Id: 38369 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:3b1c:b0:96:9ee8:5cfd with SMTP id c28csp2364863pzh; Mon, 26 Sep 2022 09:11:41 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6Dw43YhSWOrHriFUYRWPkMB4ipYizlL1jXWrgdgzLk0R5G1d4f5PH+Ld4Yy+72B+N0iFsz X-Received: by 2002:a17:906:ef8c:b0:77c:7b35:8b04 with SMTP id ze12-20020a170906ef8c00b0077c7b358b04mr18240182ejb.573.1664208700639; Mon, 26 Sep 2022 09:11:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1664208700; cv=none; d=google.com; s=arc-20160816; b=QQWLwEkK5c/c64Klp0Jb7kiZEPMiTQp7z8smYFqL0wf7B2t6DyGpXt6hV1nuU40YCh ixzoNDiELViVwhVoULbOKgzueaE3IDmlGzmwAYdQKB63EMNbPoi6uRLfN/elne+9jvl4 7smtiqQLTgr1t4U26P/KWxGvOLRQieEtmp4eqCx1cAZkb37IP8eNdmMljM7dPefkKB/G uHLFbnxbb3Y6VDiZaVTyHuBmYvWCheftpxSlwtaBWxYIQbTuZNSQzjZ6dQynEiPPLJBx qWtKj8NjSC2ugxsI/JptlLqqG0+rTpzeq9ocvbgE6k1xlVPnXncxJ8j6jXgh1eX5Zrex thAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=dMMJh7CwQ6xUCZcfQmC7wagIcuWhlk3r1Jd1p28vRe0=; b=M27zVxDJHcWvDxqoD6gUm4xsQalRDkz9ONcjL98b0KWm5P0BByVZXtbHKmft6vU+W8 ScBVQAssxlUR2AbFDs+rw7bhHdeuga8fxzTDm91883+s35UupwpGNj8oEBshjGXaw8+5 Z+dEpGN0nu/ilatsLMHcYK4HJRslLM7Ab7WjGfNRzflKrLWF0aMdQNc/08fvKlFYRuqe MgqwwXEMOOSPWVnzQfn5EijGi1SFxYyRxGHfbQMZmSjPNg9j5pfgikIiZNX7eWwZ2qxE syVpEXTnoRBIHU8BumM6UNqp2Nr1ahsh/10m3CtqwSbx3n/xN+bydQgfsPiLI4A78Bid 9Tag== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=fyNNMdu7; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dy8-20020a05640231e800b004573107a5basi4027581edb.352.2022.09.26.09.11.39; Mon, 26 Sep 2022 09:11:40 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=fyNNMdu7; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 753D468BB0C; Mon, 26 Sep 2022 19:11:35 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B87D268B7DE for ; Mon, 26 Sep 2022 19:11:28 +0300 (EEST) Received: by mail-pl1-f173.google.com with SMTP id b21so6667613plz.7 for ; Mon, 26 Sep 2022 09:11:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date; bh=1hzSaECtdxj2cgEK37rNHRQI1W/EAqZbeV/g9vjgTFk=; b=fyNNMdu7sCS99iEfs1/fUeHMN9Eqj7HHoFSLCCmjv6/857BXGD5Tzmnwurv0YKGG+a qpN60K6fis+RX+IFYg3BUbmi67Xqzlk1Nis7ANcHLRE2vT8bVRUA58EbotcqlKG8JNQo VDPT/jh04gHItu3LAdbmIuAkcrM0mnYmf2xplhfjtz7bZXpoeOdxOU0C3RJSvHMhSDb8 +VjrPQqWUA8+5t+vWEUpdg9KxM49IwGiifyUpPUE+gC5Iixb9wgWDembHIdFNUv7pcnX 2KE9i3wZJ4g7JrFuqEBpB0ocvGjRDrDmPintFVzLm9RqIrUzAxiiYWznxwGsDWjVn0TG GV1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date; bh=1hzSaECtdxj2cgEK37rNHRQI1W/EAqZbeV/g9vjgTFk=; b=YsWKyGW6RXbDFql4VJ8fNSZx/PZ6HIV+oLg95BpnG4nb1+yB/5/iSVzUFzBqecP/EX ucF2PnkBZwyJeKDsFxDNAc6noqEfbAKjSQj4qq0bupVUP97m/mPo3vU2cN5/4xXRCSxu lf8TgrctUJFl36ZLSMUi/e2L5oa3xw3UXSTXIoU+AJHsI2ANDJ3d4wtEoEZqdqAE+jPm 3stwqgIyMUkf8nCB9y19sxs1OaOC3jk53DX/JoK0QbgC+FIlsiwM2fjIBqN+XFJbkdWC 4zf/e3/M6xVj1LuZOoghnTiz9l5Psrq0bPRx3J8WpEmhau93lH8h6+d+JHZD/tijOYaw lT7g== X-Gm-Message-State: ACrzQf3fpa64aa940U8gS5K0yJugoTjA6PxPp0Ok03tGv0bdssvNrA4J cA/nq1ZpEqQ1g11RERgOuqjaR0tq661gjQ== X-Received: by 2002:a17:902:904c:b0:178:afa2:43f5 with SMTP id w12-20020a170902904c00b00178afa243f5mr23375750plz.78.1664208686374; Mon, 26 Sep 2022 09:11:26 -0700 (PDT) Received: from wilferico.thefacebook.com ([2620:10d:c090:400::5:80bc]) by smtp.gmail.com with ESMTPSA id s13-20020a170902ea0d00b0016d72804664sm11391541plg.205.2022.09.26.09.11.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Sep 2022 09:11:26 -0700 (PDT) From: Chema Gonzalez To: ffmpeg-devel@ffmpeg.org Date: Mon, 26 Sep 2022 09:11:22 -0700 Message-Id: <20220926161122.1352372-1-chemag@gmail.com> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/1] libswscale: force a minimum size of the slide for bayer sources X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Chema Gonzalez , anton@khirnov.net Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Ym7Cozx1zo/6 Bayer sources are read in groups of 2 lines (e.g. for a BGGR flavor, the first row contains only B and G samples, while the second row contains only G and R samples). They need to be read as a whole. Tested: `` $ echo -ne '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' > image.raw $ xxd image.raw 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000030: ffff ffff ffff ffff ffff ffff ffff ffff ................ ``` And then: ``` $ ./ffmpeg -y -f rawvideo -pixel_format bayer_bggr8 -s 8x8 \ -i image.raw -f rawvideo -pix_fmt rgb24 \ -video_size 8x8 image.raw.rgb ... Assertion srcSliceH > 1 failed at libswscale/swscale_unscaled.c:1310 Aborted (core dumped)ated 2 times ``` We can see that the issue relates to the ffmpeg parallelization. ``` $ ffmpeg -y -filter_threads 1 -f rawvideo -pixel_format bayer_bggr8 \ -s 8x8 -i image.raw -f rawvideo -pix_fmt rgb24 \ -video_size 8x8 image.raw.rgb ... frame= 1 fps=0.0 q=-0.0 Lsize= 0kB time=00:00:00.00 bitrate=N/A speed= 0x eed=N/A video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% $ xxd image.raw.rgb 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000050: 7f00 3f7f 0000 7f00 3f7f 0000 0000 0000 ..?.....?....... 00000060: ffff ffff ffff 7fbf ff7f ffff 7fbf ff7f ................ 00000070: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000080: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000090: ffff ffff ffff ffff ffff ffff ffff ffff ................ 000000a0: ffff ffff ffff ffff ffff ffff ffff ffff ................ 000000b0: ffff ffff ffff ffff ffff ffff ffff ffff ................ ``` Problem seems to be that `ff_sws_slice_worker()` [libswscale/swscale.c:1222] tries to slice the input to parallelize the scaling task, in my case in 16 different jobs (gdb'ing the process shows `nb_threads == nb_jobs == 16`). The 8x8 input is therefore divided in eight 8x1 slices (1-pixel height), which eventually breaks in `bayer_to_rgb24_wrapper()` as it asserts `srcSliceH > 1`. The problem is the same in the 3 Bayer conversion functions (`bayer_to_rgb24_wrapper()`, `bayer_to_rgb48_wrapper()`, and `bayer_to_yv12_wrapper()`. The solution was suggested by Anton Khirnov. We set the `dst_slice_align` value to 2 for Bayer conversions. ``` $ ./ffmpeg -y -f rawvideo -pixel_format bayer_bggr8 -s 8x8 \ -i image.raw -f rawvideo -pix_fmt rgb24 \ -video_size 8x8 image.raw.rgb ... frame= 1 fps=0.0 q=-0.0 Lsize= 0kB time=00:00:00.00 bitrate=N/A speed= 0x eed=N/A video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% $ xxd image.raw.rgb 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000060: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000070: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000080: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000090: ffff ffff ffff ffff ffff ffff ffff ffff ................ 000000a0: ffff ffff ffff ffff ffff ffff ffff ffff ................ 000000b0: ffff ffff ffff ffff ffff ffff ffff ffff ................ ``` We can see the slicing at work, though: The demosaicing does not carry through different slices (workers). Compare to forcing a single worker: ``` $ ./ffmpeg -y -filter_threads 1 -f rawvideo -pixel_format bayer_bggr8 \ -s 8x8 -i image.raw -f rawvideo -pix_fmt rgb24 \ -video_size 8x8 image.raw.alt.rgb ... frame= 1 fps=0.0 q=-0.0 Lsize= 0kB time=00:00:00.00 bitrate=N/A speed= 0x eed=N/A video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% $ xxd image.raw.rgb 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ $ xxd /tmp/image.raw.alt.rgb 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000050: 7f00 3f7f 0000 7f00 3f7f 0000 0000 0000 ..?.....?....... 00000060: ffff ffff ffff 7fbf ff7f ffff 7fbf ff7f ................ 00000070: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000080: ffff ffff ffff ffff ffff ffff ffff ffff ................ 00000090: ffff ffff ffff ffff ffff ffff ffff ffff ................ 000000a0: ffff ffff ffff ffff ffff ffff ffff ffff ................ 000000b0: ffff ffff ffff ffff ffff ffff ffff ffff ................ ``` --- libswscale/swscale_unscaled.c | 1 + 1 file changed, 1 insertion(+) diff --git a/libswscale/swscale_unscaled.c b/libswscale/swscale_unscaled.c index 8838cc8b53..9af2e7ecc3 100644 --- a/libswscale/swscale_unscaled.c +++ b/libswscale/swscale_unscaled.c @@ -2095,6 +2095,7 @@ void ff_get_unscaled_swscale(SwsContext *c) c->convert_unscaled = rgbToPlanarRgbWrapper; if (isBayer(srcFormat)) { + c->dst_slice_align = 2; if (dstFormat == AV_PIX_FMT_RGB24) c->convert_unscaled = bayer_to_rgb24_wrapper; else if (dstFormat == AV_PIX_FMT_RGB48)