From patchwork Wed Sep 20 09:01:59 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tobias Rapp X-Patchwork-Id: 5204 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp506967jaa; Wed, 20 Sep 2017 02:02:58 -0700 (PDT) X-Google-Smtp-Source: AOwi7QD05sdTWcE/WbHME4TWQABYyfEOULmTLvZGYoDTp8bbbVcYfW9yz501y4KCe/7g/gxIEflu X-Received: by 10.28.113.200 with SMTP id d69mr3578804wmi.152.1505898178231; Wed, 20 Sep 2017 02:02:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505898178; cv=none; d=google.com; s=arc-20160816; b=efnPwgAw7blCV5MK24ZzxXVzHaoQcWCZEGO9hIxCXn6ZZ5qXpBE/gadsUZyL46tmvS TL7o8itKD/6K170kPm4C7MshIesiJRwL/D+FRhodbgWtY9I8i/3Nguf3azjefbzzapcr SyykwSJWjvW9qtywwYKR/9/GarbMCMJzU0+k7mJ2T151lJJPR8mMKQAtKAFLZVoo/jVl CwHJTSg68bKcOeO3nbhYqYLXn8u2lJ5hiejXItDIbBO/ovxLVNEwRR70l8i0tyMllimz /XUmREP/4Ic1TafSVGA3UcHPmsg9WhmzxUAzisNHrnJ8AxYQf8BgcBoOjDOunPWDkMob QU1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to:arc-authentication-results; bh=GCqKHXUdRyUxi1QwQWxDdns7yLsmOdpdIghdtxB54DA=; b=pkXc3U2iUTbPUx694I7TNIufxe3RcHriZvh8vqLvZdnCBiqRM9c06fP2wEW65uD/Bv 2O01084vh+HqmEqNazoFuwZ9ICUCSmJ06PxFOpyBz3+gDQIvJO26JwT1a6euxDd83Yid p+7lE9OUv8jAQP5R6bjwvZ/3wCWZgioueLpmiNP0tGhehbJcXrzyvoaprnhMJOLu4BFS WjnC95BeSFjmpBpThQCy0BaekaAEddNFUUpD8SHYW3VE2bb9Oc0PQf8luJBFBvNi+GOc Rky59y+wf+ew9UGQz4OUZIpBEIuzA0yFa2oc2rzwYVS5qCoR/U56KJbrjELAm9FETuy5 1elg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e13si1030023wmf.131.2017.09.20.02.02.56; Wed, 20 Sep 2017 02:02:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CA160688287; Wed, 20 Sep 2017 12:02:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mx01.mail.netstorage.at (mx01.mail.netstorage.at [89.207.144.13]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C1B67680354 for ; Wed, 20 Sep 2017 12:02:39 +0300 (EEST) Received: from p1002.netstorage.at (p1002.netstorage.at [89.207.146.186]) by mx01.mail.netstorage.at (Postfix) with ESMTPS id 6B06BA2923 for ; Wed, 20 Sep 2017 11:02:00 +0200 (CEST) Received: from mailix (noaport.de [46.237.252.213]) by p1002.netstorage.at (Postfix) with ESMTPA id 19FFB82A1F for ; Wed, 20 Sep 2017 11:02:00 +0200 (CEST) Received: from frogstar-a.kuhnle.local (frogstar-a.kuhnle.local [192.168.0.26]) by mailix with ESMTPA ; Wed, 20 Sep 2017 11:02:00 +0200 From: Tobias Rapp To: ffmpeg-devel@ffmpeg.org Date: Wed, 20 Sep 2017 11:01:59 +0200 Message-Id: <1505898119-2956-1-git-send-email-t.rapp@noa-archive.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <20170919233843.GG7094@nb4> References: <20170919233843.GG7094@nb4> X-PPP-Message-ID: <20170920090200.27098.77511@p1002.netstorage.at> X-PPP-Vhost: noa-archive.com X-NetStorage-MailScanner-Information: Please contact the ISP for more information X-NetStorage-MailScanner-ID: 6B06BA2923.A594E X-NetStorage-MailScanner: Found to be clean X-NetStorage-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (nicht zwischen gespeichert, Wertung=-0.5, benoetigt 6, autolearn=not spam, BAYES_00 -0.50) X-NetStorage-MailScanner-From: t.rapp@noa-archive.com X-Spam-Status: No Subject: [FFmpeg-devel] [PATCH v4] fate: add tests for psnr and ssim filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Metadata filter output is passed through an Awk script comparing floats against reference values with specified "fuzz" tolerance to account for architectural differences (e.g. x86-32 vs. x86-64). Signed-off-by: Tobias Rapp --- tested on Linux x86-32/64 and Mips (Qemu) v4: - fixed iteration order of final print loop - update of debug output v3: - avoid precision loss due to rounding of float values v2: - removed CPUFLAGS work-around for ssim filter issue - added metadata float value post-processing script tests/fate-run.sh | 9 +++++ tests/fate/filter-video.mak | 14 ++++++++ tests/ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++ tests/ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++ tests/ref/fate/filter-refcmp-ssim-rgb | 30 ++++++++++++++++ tests/ref/fate/filter-refcmp-ssim-yuv | 30 ++++++++++++++++ tests/refcmp-metadata.awk | 64 +++++++++++++++++++++++++++++++++++ 7 files changed, 237 insertions(+) create mode 100644 tests/ref/fate/filter-refcmp-psnr-rgb create mode 100644 tests/ref/fate/filter-refcmp-psnr-yuv create mode 100644 tests/ref/fate/filter-refcmp-ssim-rgb create mode 100644 tests/ref/fate/filter-refcmp-ssim-yuv create mode 100644 tests/refcmp-metadata.awk diff --git a/tests/fate-run.sh b/tests/fate-run.sh index 9aa9a22..c5480c7 100755 --- a/tests/fate-run.sh +++ b/tests/fate-run.sh @@ -234,6 +234,15 @@ lavftest(){ ${base}/lavf-regression.sh $t lavf tests/vsynth1 "$target_exec" "$target_path" "$threads" "$thread_type" "$cpuflags" "$target_samples" } +refcmp_metadata(){ + refcmp=$1 + pixfmt=$2 + fuzz=${3:-0.001} + ffmpeg $FLAGS $ENC_OPTS \ + -lavfi "testsrc2=size=300x200:rate=1:duration=5,format=${pixfmt},split[ref][tmp];[tmp]avgblur=4[enc];[enc][ref]${refcmp},metadata=print:file=-" \ + -f null /dev/null | awk -v ref=${ref} -v fuzz=${fuzz} -f ${base}/refcmp-metadata.awk - +} + video_filter(){ filters=$1 shift diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index d1e1341..78cd471 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -747,6 +747,20 @@ FATE_FILTER_SAMPLES-$(call ALLYES, MOV_DEMUXER H264_DECODER AAC_FIXED_DECODER PC fate-filter-meta-4560-rotate0: tests/data/file4560-override2rotate0.mov fate-filter-meta-4560-rotate0: CMD = framecrc -flags +bitexact -c:a aac_fixed -i $(TARGET_PATH)/tests/data/file4560-override2rotate0.mov +REFCMP_DEPS = FFMPEG LAVFI_INDEV TESTSRC2_FILTER AVGBLUR_FILTER METADATA_FILTER + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) PSNR_FILTER) += fate-filter-refcmp-psnr-rgb +fate-filter-refcmp-psnr-rgb: CMD = refcmp_metadata psnr rgb24 + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) PSNR_FILTER) += fate-filter-refcmp-psnr-yuv +fate-filter-refcmp-psnr-yuv: CMD = refcmp_metadata psnr yuv422p + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) SSIM_FILTER) += fate-filter-refcmp-ssim-rgb +fate-filter-refcmp-ssim-rgb: CMD = refcmp_metadata ssim rgb24 + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) SSIM_FILTER) += fate-filter-refcmp-ssim-yuv +fate-filter-refcmp-ssim-yuv: CMD = refcmp_metadata ssim yuv422p + FATE_SAMPLES_FFPROBE += $(FATE_METADATA_FILTER-yes) FATE_SAMPLES_FFMPEG += $(FATE_FILTER_SAMPLES-yes) FATE_FFMPEG += $(FATE_FILTER-yes) diff --git a/tests/ref/fate/filter-refcmp-psnr-rgb b/tests/ref/fate/filter-refcmp-psnr-rgb new file mode 100644 index 0000000..7d413f4 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-psnr-rgb @@ -0,0 +1,45 @@ +frame:0 pts:0 pts_time:0 +lavfi.psnr.mse.r=1381.80 +lavfi.psnr.psnr.r=16.73 +lavfi.psnr.mse.g=896.00 +lavfi.psnr.psnr.g=18.61 +lavfi.psnr.mse.b=277.38 +lavfi.psnr.psnr.b=23.70 +lavfi.psnr.mse_avg=851.73 +lavfi.psnr.psnr_avg=18.83 +frame:1 pts:1 pts_time:1 +lavfi.psnr.mse.r=1380.37 +lavfi.psnr.psnr.r=16.73 +lavfi.psnr.mse.g=975.91 +lavfi.psnr.psnr.g=18.24 +lavfi.psnr.mse.b=435.72 +lavfi.psnr.psnr.b=21.74 +lavfi.psnr.mse_avg=930.67 +lavfi.psnr.psnr_avg=18.44 +frame:2 pts:2 pts_time:2 +lavfi.psnr.mse.r=1403.20 +lavfi.psnr.psnr.r=16.66 +lavfi.psnr.mse.g=954.05 +lavfi.psnr.psnr.g=18.34 +lavfi.psnr.mse.b=494.22 +lavfi.psnr.psnr.b=21.19 +lavfi.psnr.mse_avg=950.49 +lavfi.psnr.psnr_avg=18.35 +frame:3 pts:3 pts_time:3 +lavfi.psnr.mse.r=1452.80 +lavfi.psnr.psnr.r=16.51 +lavfi.psnr.mse.g=1001.02 +lavfi.psnr.psnr.g=18.13 +lavfi.psnr.mse.b=557.39 +lavfi.psnr.psnr.b=20.67 +lavfi.psnr.mse_avg=1003.74 +lavfi.psnr.psnr_avg=18.11 +frame:4 pts:4 pts_time:4 +lavfi.psnr.mse.r=1401.25 +lavfi.psnr.psnr.r=16.67 +lavfi.psnr.mse.g=1009.80 +lavfi.psnr.psnr.g=18.09 +lavfi.psnr.mse.b=602.42 +lavfi.psnr.psnr.b=20.33 +lavfi.psnr.mse_avg=1004.49 +lavfi.psnr.psnr_avg=18.11 diff --git a/tests/ref/fate/filter-refcmp-psnr-yuv b/tests/ref/fate/filter-refcmp-psnr-yuv new file mode 100644 index 0000000..942542a --- /dev/null +++ b/tests/ref/fate/filter-refcmp-psnr-yuv @@ -0,0 +1,45 @@ +frame:0 pts:0 pts_time:0 +lavfi.psnr.mse.y=222.06 +lavfi.psnr.psnr.y=24.67 +lavfi.psnr.mse.u=339.38 +lavfi.psnr.psnr.u=22.82 +lavfi.psnr.mse.v=705.41 +lavfi.psnr.psnr.v=19.65 +lavfi.psnr.mse_avg=372.23 +lavfi.psnr.psnr_avg=22.42 +frame:1 pts:1 pts_time:1 +lavfi.psnr.mse.y=236.74 +lavfi.psnr.psnr.y=24.39 +lavfi.psnr.mse.u=416.17 +lavfi.psnr.psnr.u=21.94 +lavfi.psnr.mse.v=704.98 +lavfi.psnr.psnr.v=19.65 +lavfi.psnr.mse_avg=398.66 +lavfi.psnr.psnr_avg=22.12 +frame:2 pts:2 pts_time:2 +lavfi.psnr.mse.y=234.79 +lavfi.psnr.psnr.y=24.42 +lavfi.psnr.mse.u=435.72 +lavfi.psnr.psnr.u=21.74 +lavfi.psnr.mse.v=699.60 +lavfi.psnr.psnr.v=19.68 +lavfi.psnr.mse_avg=401.23 +lavfi.psnr.psnr_avg=22.10 +frame:3 pts:3 pts_time:3 +lavfi.psnr.mse.y=250.88 +lavfi.psnr.psnr.y=24.14 +lavfi.psnr.mse.u=479.73 +lavfi.psnr.psnr.u=21.32 +lavfi.psnr.mse.v=707.55 +lavfi.psnr.psnr.v=19.63 +lavfi.psnr.mse_avg=422.26 +lavfi.psnr.psnr_avg=21.88 +frame:4 pts:4 pts_time:4 +lavfi.psnr.mse.y=241.05 +lavfi.psnr.psnr.y=24.31 +lavfi.psnr.mse.u=505.04 +lavfi.psnr.psnr.u=21.10 +lavfi.psnr.mse.v=716.00 +lavfi.psnr.psnr.v=19.58 +lavfi.psnr.mse_avg=425.79 +lavfi.psnr.psnr_avg=21.84 diff --git a/tests/ref/fate/filter-refcmp-ssim-rgb b/tests/ref/fate/filter-refcmp-ssim-rgb new file mode 100644 index 0000000..4a06eb4 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-ssim-rgb @@ -0,0 +1,30 @@ +frame:0 pts:0 pts_time:0 +lavfi.ssim.R=0.72 +lavfi.ssim.G=0.76 +lavfi.ssim.B=0.89 +lavfi.ssim.All=0.79 +lavfi.ssim.dB=6.74 +frame:1 pts:1 pts_time:1 +lavfi.ssim.R=0.70 +lavfi.ssim.G=0.74 +lavfi.ssim.B=0.85 +lavfi.ssim.All=0.77 +lavfi.ssim.dB=6.31 +frame:2 pts:2 pts_time:2 +lavfi.ssim.R=0.71 +lavfi.ssim.G=0.75 +lavfi.ssim.B=0.84 +lavfi.ssim.All=0.76 +lavfi.ssim.dB=6.29 +frame:3 pts:3 pts_time:3 +lavfi.ssim.R=0.70 +lavfi.ssim.G=0.73 +lavfi.ssim.B=0.83 +lavfi.ssim.All=0.76 +lavfi.ssim.dB=6.11 +frame:4 pts:4 pts_time:4 +lavfi.ssim.R=0.71 +lavfi.ssim.G=0.74 +lavfi.ssim.B=0.80 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=6.05 diff --git a/tests/ref/fate/filter-refcmp-ssim-yuv b/tests/ref/fate/filter-refcmp-ssim-yuv new file mode 100644 index 0000000..0cd46c8 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-ssim-yuv @@ -0,0 +1,30 @@ +frame:0 pts:0 pts_time:0 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.76 +lavfi.ssim.V=0.69 +lavfi.ssim.All=0.76 +lavfi.ssim.dB=6.25 +frame:1 pts:1 pts_time:1 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.73 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=6.08 +frame:2 pts:2 pts_time:2 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.73 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=6.10 +frame:3 pts:3 pts_time:3 +lavfi.ssim.Y=0.79 +lavfi.ssim.U=0.72 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=5.94 +frame:4 pts:4 pts_time:4 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.72 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=5.97 diff --git a/tests/refcmp-metadata.awk b/tests/refcmp-metadata.awk new file mode 100644 index 0000000..fa21aad --- /dev/null +++ b/tests/refcmp-metadata.awk @@ -0,0 +1,64 @@ +# Compare metadata filter output containing float value strings to reference +# output data. Returns the whole reference data if delta of each value is below +# threshold, else returns the whole input data. + +function abs(val) { + return ((val < 0.0) ? -val : val); +} + +function max(val1, val2) { + return ((val1 >= val2) ? val1 : val2); +} + +function is_numeric_str(str) { + return (str ~ /^[+-]?[0-9]*\.?[0-9]+$/); +} + +BEGIN { + FS = "="; + # check for "fuzz" (threshold) program parameter, else use default + if (fuzz <= 0.0) { + fuzz = 0.1; + } + # check for "ref" (reference file) program parameter + if (ref) { + ref_nr = 0; + while ((getline line < ref) > 0) { + ref_nr++; + ref_lines[ref_nr] = line; + if (split(line, fields) == 2 && is_numeric_str(fields[2])) { + ref_keys[ref_nr] = fields[1]; + ref_vals[ref_nr] = fields[2] + 0; # convert to number + } + } + close(ref); + } + delta_max = 0; + result = (ref ? 1 : 0); +} + +{ + cmp_lines[NR] = $0; + if (NF == 2 && is_numeric_str($2) && ref_vals[NR]) { + val = $2 + 0; # convert to number + delta = abs((val / ref_vals[NR]) - 1); + delta_max = max(delta_max, delta); + result = result && ($1 == ref_keys[NR]) && (delta <= fuzz); + } else { + result = result && ($0 == ref_lines[NR]); + } +} + +END { + if (result) { + for (i = 1; i <= ref_nr; i++) + print ref_lines[i]; + } else { + for (i = 1; i <= NR; i++) + print cmp_lines[i]; + if (NR != ref_nr) + print "[refcmp] lines: " NR " != " ref_nr > "/dev/stderr"; + if (delta_max >= fuzz) + print "[refcmp] delta_max: " delta_max " >= " fuzz > "/dev/stderr"; + } +}