From patchwork Tue Sep 19 23:38:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 5195 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp117939jaa; Tue, 19 Sep 2017 16:40:55 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBn/FpXFAXroHxjGLDon2KPfqpbzAFGjeFlxkVWc8D10y8/qN5cLq2UGr9RzRFyVuXV8sI6 X-Received: by 10.28.238.218 with SMTP id j87mr2446486wmi.44.1505864455185; Tue, 19 Sep 2017 16:40:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505864455; cv=none; d=google.com; s=arc-20160816; b=ujWM0YNBkbmsRjUDTTkKB34YFnCCWw0n2UEQQVY3vHTSdRpfkghSxpUgz6n1UkzWaf pzS8MpUtdJiNh73uHCLY+oWlL7iiL8hvErHUFISaQ8Obi9sEk5VKkAhC/EDVx8hrckWK AJbHkYim3Oc+OpcR2nkDgzphmsAYBfOBsZgmdj/qCZFu70lSi0n6omqDzPMxkWdrYgI4 sg/1y3oXImmPL4qFHHKyfJpu3zXJpTLVWH+KcwnR+S/3QsX36F99LnfRQjGQ7Du/KF49 /yDqZr/qK/hJi7vzKZVuRaQ6bNqkgXVY5+Hf8Q5xVJwjSzSiBZz7XiU9Dvl9rRTgq9Wk bciA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:user-agent :in-reply-to:mime-version:references:message-id:to:from:date :delivered-to:arc-authentication-results; bh=dd5M1AqMYedEQh9kuaqVcF/33MluFLLDnChMMJh/Pp4=; b=HJsxhM/j79RrGSiqy7Ehov0F+auNa5RDnL/UrR6SkBhSPY4BO8j2Vx6PCOVr9qXNT6 f3cQUzljm2D6YfsIkPzuxXnh2G7GxKNVZP333FNgKmAoCDqyN68yz1D+2IqhN9ZSatT7 CpPxCKq3ApV+0oFUHZAf1MwZ/hhBUpcJkTcjAqCqAoJp2/bGejCNCveI8dn0mIHb7J3F QuSf/hOi7BAc4pDKFKq4e4Q3UX1jx1fjzj86I7+W0sADUKvTCf4R3H3/zVk7+3JPbYwX CApO/zUrZmghLKF1DaaMUQHuVW/C7YF+vqABxXjF2w+ktWX+JabcBSnY1tC8XslI/tuN MUdw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p5si313270wmb.175.2017.09.19.16.40.53; Tue, 19 Sep 2017 16:40:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A7140689203; Wed, 20 Sep 2017 02:40:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B70D368837A for ; Wed, 20 Sep 2017 02:40:37 +0300 (EEST) X-Originating-IP: 213.47.41.20 Received: from localhost (213-47-41-20.cable.dynamic.surfer.at [213.47.41.20]) (Authenticated sender: michael@niedermayer.cc) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 6710541C08A for ; Wed, 20 Sep 2017 01:40:46 +0200 (CEST) Date: Wed, 20 Sep 2017 01:38:43 +0200 From: Michael Niedermayer To: FFmpeg development discussions and patches Message-ID: <20170919233843.GG7094@nb4> References: <1504087412-14006-1-git-send-email-t.rapp@noa-archive.com> <1505807325-9884-1-git-send-email-t.rapp@noa-archive.com> MIME-Version: 1.0 In-Reply-To: <1505807325-9884-1-git-send-email-t.rapp@noa-archive.com> User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: [FFmpeg-devel] [PATCH v3] fate: add tests for psnr and ssim filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On Tue, Sep 19, 2017 at 09:48:45AM +0200, Tobias Rapp wrote: > Metadata filter output is passed through an Awk script comparing floats > against reference values with specified "fuzz" tolerance to account for > architectural differences (e.g. x86-32 vs. x86-64). > > Signed-off-by: Tobias Rapp > --- > tested on Linux x86-32/64 and Mips (Qemu) > > v3: > - avoid precision loss due to rounding of float values > > v2: > - removed CPUFLAGS work-around for ssim filter issue > - added metadata float value post-processing script > > tests/fate-run.sh | 9 ++++++ > tests/fate/filter-video.mak | 14 ++++++++ > tests/ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++ > tests/ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++ > tests/ref/fate/filter-refcmp-ssim-rgb | 30 +++++++++++++++++ > tests/ref/fate/filter-refcmp-ssim-yuv | 30 +++++++++++++++++ > tests/refcmp-metadata.awk | 61 +++++++++++++++++++++++++++++++++++ > 7 files changed, 234 insertions(+) > create mode 100644 tests/ref/fate/filter-refcmp-psnr-rgb > create mode 100644 tests/ref/fate/filter-refcmp-psnr-yuv > create mode 100644 tests/ref/fate/filter-refcmp-ssim-rgb > create mode 100644 tests/ref/fate/filter-refcmp-ssim-yuv > create mode 100644 tests/refcmp-metadata.awk This seems not to work (normal x86-64 linux) but maybe ive done something silly Test filter-refcmp-ssim-yuv failed. Look at tests/data/fate/filter-refcmp-ssim-yuv.err for details. make: *** [fate-filter-refcmp-ssim-yuv] Error 1 [...] --- ./tests/ref/fate/filter-refcmp-psnr-rgb 2017-09-20 00:58:04.063337500 +0200 +++ tests/data/fate/filter-refcmp-psnr-rgb 2017-09-20 00:59:26.399339234 +0200 @@ -1,12 +1,3 @@ -frame:0 pts:0 pts_time:0 -lavfi.psnr.mse.r=1381.80 -lavfi.psnr.psnr.r=16.73 -lavfi.psnr.mse.g=896.00 -lavfi.psnr.psnr.g=18.61 -lavfi.psnr.mse.b=277.38 -lavfi.psnr.psnr.b=23.70 -lavfi.psnr.mse_avg=851.73 -lavfi.psnr.psnr_avg=18.83 frame:1 pts:1 pts_time:1 lavfi.psnr.mse.r=1380.37 lavfi.psnr.psnr.r=16.73 @@ -14,22 +5,12 @@ lavfi.psnr.psnr.g=18.24 lavfi.psnr.mse.b=435.72 lavfi.psnr.psnr.b=21.74 -lavfi.psnr.mse_avg=930.67 -lavfi.psnr.psnr_avg=18.44 -frame:2 pts:2 pts_time:2 -lavfi.psnr.mse.r=1403.20 -lavfi.psnr.psnr.r=16.66 -lavfi.psnr.mse.g=954.05 -lavfi.psnr.psnr.g=18.34 -lavfi.psnr.mse.b=494.22 -lavfi.psnr.psnr.b=21.19 -lavfi.psnr.mse_avg=950.49 -lavfi.psnr.psnr_avg=18.35 -frame:3 pts:3 pts_time:3 -lavfi.psnr.mse.r=1452.80 lavfi.psnr.psnr.r=16.51 +lavfi.psnr.mse_avg=930.67 lavfi.psnr.mse.g=1001.02 +lavfi.psnr.psnr_avg=18.44 lavfi.psnr.psnr.g=18.13 +frame:2 pts:2 pts_time:2 lavfi.psnr.mse.b=557.39 lavfi.psnr.psnr.b=20.67 lavfi.psnr.mse_avg=1003.74 @@ -37,9 +18,28 @@ frame:4 pts:4 pts_time:4 lavfi.psnr.mse.r=1401.25 lavfi.psnr.psnr.r=16.67 +frame:0 pts:0 pts_time:0 +lavfi.psnr.mse.r=1381.80 +lavfi.psnr.psnr.r=16.73 +lavfi.psnr.mse.g=896.00 +lavfi.psnr.psnr.g=18.61 +lavfi.psnr.mse.b=277.38 +lavfi.psnr.psnr.b=23.70 +lavfi.psnr.mse_avg=851.73 +lavfi.psnr.psnr_avg=18.83 +lavfi.psnr.mse.r=1403.20 +lavfi.psnr.psnr.r=16.66 +lavfi.psnr.mse.g=954.05 +lavfi.psnr.psnr.g=18.34 +lavfi.psnr.mse.b=494.22 +lavfi.psnr.psnr.b=21.19 +lavfi.psnr.mse_avg=950.49 lavfi.psnr.mse.g=1009.80 +lavfi.psnr.psnr_avg=18.35 lavfi.psnr.psnr.g=18.09 +frame:3 pts:3 pts_time:3 lavfi.psnr.mse.b=602.42 +lavfi.psnr.mse.r=1452.80 lavfi.psnr.psnr.b=20.33 lavfi.psnr.mse_avg=1004.49 lavfi.psnr.psnr_avg=18.11 Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details. make: *** [fate-filter-refcmp-psnr-rgb] Error 1 TEST fitsdec-bitpix-32 --- ./tests/ref/fate/filter-refcmp-ssim-rgb 2017-09-20 00:58:04.063337500 +0200 +++ tests/data/fate/filter-refcmp-ssim-rgb 2017-09-20 00:59:26.415339235 +0200 @@ -1,30 +1,30 @@ -frame:0 pts:0 pts_time:0 -lavfi.ssim.R=0.72 -lavfi.ssim.G=0.76 +lavfi.ssim.R=0.71 +lavfi.ssim.All=0.76 lavfi.ssim.B=0.89 +lavfi.ssim.G=0.74 +lavfi.ssim.dB=6.29 lavfi.ssim.All=0.79 +lavfi.ssim.B=0.80 +frame:3 pts:3 pts_time:3 lavfi.ssim.dB=6.74 +lavfi.ssim.All=0.75 frame:1 pts:1 pts_time:1 lavfi.ssim.R=0.70 lavfi.ssim.G=0.74 lavfi.ssim.B=0.85 -lavfi.ssim.All=0.77 -lavfi.ssim.dB=6.31 -frame:2 pts:2 pts_time:2 -lavfi.ssim.R=0.71 -lavfi.ssim.G=0.75 -lavfi.ssim.B=0.84 -lavfi.ssim.All=0.76 -lavfi.ssim.dB=6.29 -frame:3 pts:3 pts_time:3 lavfi.ssim.R=0.70 +lavfi.ssim.All=0.77 +lavfi.ssim.dB=6.05 lavfi.ssim.G=0.73 +lavfi.ssim.dB=6.31 lavfi.ssim.B=0.83 +frame:2 pts:2 pts_time:2 lavfi.ssim.All=0.76 +lavfi.ssim.R=0.71 +frame:0 pts:0 pts_time:0 lavfi.ssim.dB=6.11 +lavfi.ssim.G=0.75 +lavfi.ssim.R=0.72 frame:4 pts:4 pts_time:4 -lavfi.ssim.R=0.71 -lavfi.ssim.G=0.74 -lavfi.ssim.B=0.80 -lavfi.ssim.All=0.75 -lavfi.ssim.dB=6.05 +lavfi.ssim.B=0.84 +lavfi.ssim.G=0.76 Test filter-refcmp-ssim-rgb failed. Look at tests/data/fate/filter-refcmp-ssim-rgb.err for details. make: *** [fate-filter-refcmp-ssim-rgb] Error 1 TEST fitsdec-bitpix-64 --- ./tests/ref/fate/filter-refcmp-psnr-yuv 2017-09-20 00:58:04.063337500 +0200 +++ tests/data/fate/filter-refcmp-psnr-yuv 2017-09-20 00:59:26.435339235 +0200 @@ -1,12 +1,3 @@ -frame:0 pts:0 pts_time:0 -lavfi.psnr.mse.y=222.06 -lavfi.psnr.psnr.y=24.67 -lavfi.psnr.mse.u=339.38 -lavfi.psnr.psnr.u=22.82 -lavfi.psnr.mse.v=705.41 -lavfi.psnr.psnr.v=19.65 -lavfi.psnr.mse_avg=372.23 -lavfi.psnr.psnr_avg=22.42 frame:1 pts:1 pts_time:1 lavfi.psnr.mse.y=236.74 lavfi.psnr.psnr.y=24.39 @@ -14,22 +5,12 @@ lavfi.psnr.psnr.u=21.94 lavfi.psnr.mse.v=704.98 lavfi.psnr.psnr.v=19.65 -lavfi.psnr.mse_avg=398.66 -lavfi.psnr.psnr_avg=22.12 -frame:2 pts:2 pts_time:2 -lavfi.psnr.mse.y=234.79 -lavfi.psnr.psnr.y=24.42 -lavfi.psnr.mse.u=435.72 -lavfi.psnr.psnr.u=21.74 -lavfi.psnr.mse.v=699.60 -lavfi.psnr.psnr.v=19.68 -lavfi.psnr.mse_avg=401.23 -lavfi.psnr.psnr_avg=22.10 -frame:3 pts:3 pts_time:3 -lavfi.psnr.mse.y=250.88 lavfi.psnr.psnr.y=24.14 +lavfi.psnr.mse_avg=398.66 lavfi.psnr.mse.u=479.73 +lavfi.psnr.psnr_avg=22.12 lavfi.psnr.psnr.u=21.32 +frame:2 pts:2 pts_time:2 lavfi.psnr.mse.v=707.55 lavfi.psnr.psnr.v=19.63 lavfi.psnr.mse_avg=422.26 @@ -37,9 +18,28 @@ frame:4 pts:4 pts_time:4 lavfi.psnr.mse.y=241.05 lavfi.psnr.psnr.y=24.31 +frame:0 pts:0 pts_time:0 +lavfi.psnr.mse.y=222.06 +lavfi.psnr.psnr.y=24.67 +lavfi.psnr.mse.u=339.38 +lavfi.psnr.psnr.u=22.82 +lavfi.psnr.mse.v=705.41 +lavfi.psnr.psnr.v=19.65 +lavfi.psnr.mse_avg=372.23 +lavfi.psnr.psnr_avg=22.42 +lavfi.psnr.mse.y=234.79 +lavfi.psnr.psnr.y=24.42 +lavfi.psnr.mse.u=435.72 +lavfi.psnr.psnr.u=21.74 +lavfi.psnr.mse.v=699.60 +lavfi.psnr.psnr.v=19.68 +lavfi.psnr.mse_avg=401.23 lavfi.psnr.mse.u=505.04 +lavfi.psnr.psnr_avg=22.10 lavfi.psnr.psnr.u=21.10 +frame:3 pts:3 pts_time:3 lavfi.psnr.mse.v=716.00 +lavfi.psnr.mse.y=250.88 lavfi.psnr.psnr.v=19.58 lavfi.psnr.mse_avg=425.79 lavfi.psnr.psnr_avg=21.84 Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details. make: *** [fate-filter-refcmp-psnr-yuv] Error 1 TEST fitsdec-multi GEN tests/data/lena-gray.fits GEN tests/data/lena-gbrp.fits --- ./tests/ref/fate/filter-refcmp-ssim-yuv 2017-09-20 00:58:04.063337500 +0200 +++ tests/data/fate/filter-refcmp-ssim-yuv 2017-09-20 00:59:26.491339236 +0200 @@ -1,30 +1,30 @@ -frame:0 pts:0 pts_time:0 lavfi.ssim.Y=0.80 -lavfi.ssim.U=0.76 +lavfi.ssim.All=0.75 lavfi.ssim.V=0.69 +lavfi.ssim.U=0.72 +lavfi.ssim.dB=6.10 lavfi.ssim.All=0.76 -lavfi.ssim.dB=6.25 -frame:1 pts:1 pts_time:1 -lavfi.ssim.Y=0.80 -lavfi.ssim.U=0.73 lavfi.ssim.V=0.68 +frame:3 pts:3 pts_time:3 +lavfi.ssim.dB=6.25 lavfi.ssim.All=0.75 -lavfi.ssim.dB=6.08 -frame:2 pts:2 pts_time:2 +frame:1 pts:1 pts_time:1 lavfi.ssim.Y=0.80 lavfi.ssim.U=0.73 lavfi.ssim.V=0.68 -lavfi.ssim.All=0.75 -lavfi.ssim.dB=6.10 -frame:3 pts:3 pts_time:3 lavfi.ssim.Y=0.79 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=5.97 lavfi.ssim.U=0.72 +lavfi.ssim.dB=6.08 lavfi.ssim.V=0.68 +frame:2 pts:2 pts_time:2 lavfi.ssim.All=0.75 +lavfi.ssim.Y=0.80 +frame:0 pts:0 pts_time:0 lavfi.ssim.dB=5.94 -frame:4 pts:4 pts_time:4 +lavfi.ssim.U=0.73 lavfi.ssim.Y=0.80 -lavfi.ssim.U=0.72 +frame:4 pts:4 pts_time:4 lavfi.ssim.V=0.68 -lavfi.ssim.All=0.75 -lavfi.ssim.dB=5.97 +lavfi.ssim.U=0.76