From patchwork Tue Sep 19 07:48:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tobias Rapp X-Patchwork-Id: 5189 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp4356835jaa; Tue, 19 Sep 2017 00:49:16 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDZyZp6USXKxLMQ6Jo24iJy4WLDfwbVVDgnrtBuYxWKD+l3drK64EpGo8SW/nY8zeypw/fx X-Received: by 10.28.12.134 with SMTP id 128mr232201wmm.147.1505807356881; Tue, 19 Sep 2017 00:49:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505807356; cv=none; d=google.com; s=arc-20160816; b=IU6PKfMLB34YZ9jOGUDiQU5VufWjGKlA/qhrYR9P7PjsLreDiu+dd0SbqSBSlQYhaR eaA5hvLKogg9JTTpHBl5gKxPnrzNSOdoDTqz2/iZ+OXZopdzZeLGTUcwN7JzjWDyn+hI zU73iBtchC61tQYs/MxSog47bHgD/NM2aycmEgjgT81XxMkxpJ6gOtHmP8CAt2smJeXR Bmer2c7PuINyz0vAI60UXghu0dTBv0ccUWj6Prh6+qfZJhb+P9N4uEefDcp5IOPwaTEp hME0vrpSEaB7y7AjfQ+sZeA/I6OShkLQu78AF+PQaAh9yspN1f4j3l4BmsQgrgbrNikx d9Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to:arc-authentication-results; bh=KMISpZCLo0hXmLbb9L1Ipbux+sTYSMhEPQmA92MC1SU=; b=wCLIudE0fdgbLPkXNvVSTVGnyZTrkZqGdOqTVWoA4MJ+5PTufPTXBT7ZXWDulmaPSn ScRx0iABUKTrUy2/04ewXimB8nU6Swns5foEsr0hYdZXc8F71Vbw8ZYG43u2VLv4uola ORS+BBcqJxhPmBRhfz42vQARH6e2ZRIOG+9NcFaQYlTGVeru+IQu1u+5zWllLvlAIupf yKI+9Ewh+Peq54qTY+Vv+hUAZG77aLvw+JBsXe4EXPPlk6JUZeFD8JxqSTTdQxgoSJaG Sr2sgEH/661ZyzV1B0WDu7qmy3LIsMjjgMG7a+CXaNH4qy5/mqI18lQa6Wc33pDqmHnO N4Sg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k2si905126wmc.31.2017.09.19.00.49.15; Tue, 19 Sep 2017 00:49:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 39177688290; Tue, 19 Sep 2017 10:49:05 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mx02.mail.netstorage.at (mx02.mail.netstorage.at [89.207.146.155]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7447A6804A4 for ; Tue, 19 Sep 2017 10:48:58 +0300 (EEST) Received: from p1002.netstorage.at (p1002.netstorage.at [89.207.146.186]) by mx02.mail.netstorage.at (Postfix) with ESMTPS id 924D8A078C for ; Tue, 19 Sep 2017 09:48:46 +0200 (CEST) Received: from mailix (noaport.de [46.237.252.213]) by p1002.netstorage.at (Postfix) with ESMTPA id 5334B82873 for ; Tue, 19 Sep 2017 09:48:46 +0200 (CEST) Received: from frogstar-a.kuhnle.local (frogstar-a.kuhnle.local [192.168.0.26]) by mailix with ESMTPA ; Tue, 19 Sep 2017 09:48:46 +0200 From: Tobias Rapp To: ffmpeg-devel@ffmpeg.org Date: Tue, 19 Sep 2017 09:48:45 +0200 Message-Id: <1505807325-9884-1-git-send-email-t.rapp@noa-archive.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1504087412-14006-1-git-send-email-t.rapp@noa-archive.com> References: <1504087412-14006-1-git-send-email-t.rapp@noa-archive.com> X-PPP-Message-ID: <20170919074846.7521.47307@p1002.netstorage.at> X-PPP-Vhost: noa-archive.com X-NetStorage-MailScanner-Information: Please contact the ISP for more information X-NetStorage-MailScanner-ID: 924D8A078C.A79E4 X-NetStorage-MailScanner: Found to be clean X-NetStorage-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (nicht zwischen gespeichert, Wertung=-0.5, benoetigt 6, autolearn=not spam, BAYES_00 -0.50) X-NetStorage-MailScanner-From: t.rapp@noa-archive.com X-Spam-Status: No Subject: [FFmpeg-devel] [PATCH v3] fate: add tests for psnr and ssim filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Metadata filter output is passed through an Awk script comparing floats against reference values with specified "fuzz" tolerance to account for architectural differences (e.g. x86-32 vs. x86-64). Signed-off-by: Tobias Rapp --- tested on Linux x86-32/64 and Mips (Qemu) v3: - avoid precision loss due to rounding of float values v2: - removed CPUFLAGS work-around for ssim filter issue - added metadata float value post-processing script tests/fate-run.sh | 9 ++++++ tests/fate/filter-video.mak | 14 ++++++++ tests/ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++ tests/ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++ tests/ref/fate/filter-refcmp-ssim-rgb | 30 +++++++++++++++++ tests/ref/fate/filter-refcmp-ssim-yuv | 30 +++++++++++++++++ tests/refcmp-metadata.awk | 61 +++++++++++++++++++++++++++++++++++ 7 files changed, 234 insertions(+) create mode 100644 tests/ref/fate/filter-refcmp-psnr-rgb create mode 100644 tests/ref/fate/filter-refcmp-psnr-yuv create mode 100644 tests/ref/fate/filter-refcmp-ssim-rgb create mode 100644 tests/ref/fate/filter-refcmp-ssim-yuv create mode 100644 tests/refcmp-metadata.awk diff --git a/tests/fate-run.sh b/tests/fate-run.sh index 9aa9a22..c5480c7 100755 --- a/tests/fate-run.sh +++ b/tests/fate-run.sh @@ -234,6 +234,15 @@ lavftest(){ ${base}/lavf-regression.sh $t lavf tests/vsynth1 "$target_exec" "$target_path" "$threads" "$thread_type" "$cpuflags" "$target_samples" } +refcmp_metadata(){ + refcmp=$1 + pixfmt=$2 + fuzz=${3:-0.001} + ffmpeg $FLAGS $ENC_OPTS \ + -lavfi "testsrc2=size=300x200:rate=1:duration=5,format=${pixfmt},split[ref][tmp];[tmp]avgblur=4[enc];[enc][ref]${refcmp},metadata=print:file=-" \ + -f null /dev/null | awk -v ref=${ref} -v fuzz=${fuzz} -f ${base}/refcmp-metadata.awk - +} + video_filter(){ filters=$1 shift diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index d1e1341..78cd471 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -747,6 +747,20 @@ FATE_FILTER_SAMPLES-$(call ALLYES, MOV_DEMUXER H264_DECODER AAC_FIXED_DECODER PC fate-filter-meta-4560-rotate0: tests/data/file4560-override2rotate0.mov fate-filter-meta-4560-rotate0: CMD = framecrc -flags +bitexact -c:a aac_fixed -i $(TARGET_PATH)/tests/data/file4560-override2rotate0.mov +REFCMP_DEPS = FFMPEG LAVFI_INDEV TESTSRC2_FILTER AVGBLUR_FILTER METADATA_FILTER + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) PSNR_FILTER) += fate-filter-refcmp-psnr-rgb +fate-filter-refcmp-psnr-rgb: CMD = refcmp_metadata psnr rgb24 + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) PSNR_FILTER) += fate-filter-refcmp-psnr-yuv +fate-filter-refcmp-psnr-yuv: CMD = refcmp_metadata psnr yuv422p + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) SSIM_FILTER) += fate-filter-refcmp-ssim-rgb +fate-filter-refcmp-ssim-rgb: CMD = refcmp_metadata ssim rgb24 + +FATE_FILTER_SAMPLES-$(call ALLYES, $(REFCMP_DEPS) SSIM_FILTER) += fate-filter-refcmp-ssim-yuv +fate-filter-refcmp-ssim-yuv: CMD = refcmp_metadata ssim yuv422p + FATE_SAMPLES_FFPROBE += $(FATE_METADATA_FILTER-yes) FATE_SAMPLES_FFMPEG += $(FATE_FILTER_SAMPLES-yes) FATE_FFMPEG += $(FATE_FILTER-yes) diff --git a/tests/ref/fate/filter-refcmp-psnr-rgb b/tests/ref/fate/filter-refcmp-psnr-rgb new file mode 100644 index 0000000..7d413f4 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-psnr-rgb @@ -0,0 +1,45 @@ +frame:0 pts:0 pts_time:0 +lavfi.psnr.mse.r=1381.80 +lavfi.psnr.psnr.r=16.73 +lavfi.psnr.mse.g=896.00 +lavfi.psnr.psnr.g=18.61 +lavfi.psnr.mse.b=277.38 +lavfi.psnr.psnr.b=23.70 +lavfi.psnr.mse_avg=851.73 +lavfi.psnr.psnr_avg=18.83 +frame:1 pts:1 pts_time:1 +lavfi.psnr.mse.r=1380.37 +lavfi.psnr.psnr.r=16.73 +lavfi.psnr.mse.g=975.91 +lavfi.psnr.psnr.g=18.24 +lavfi.psnr.mse.b=435.72 +lavfi.psnr.psnr.b=21.74 +lavfi.psnr.mse_avg=930.67 +lavfi.psnr.psnr_avg=18.44 +frame:2 pts:2 pts_time:2 +lavfi.psnr.mse.r=1403.20 +lavfi.psnr.psnr.r=16.66 +lavfi.psnr.mse.g=954.05 +lavfi.psnr.psnr.g=18.34 +lavfi.psnr.mse.b=494.22 +lavfi.psnr.psnr.b=21.19 +lavfi.psnr.mse_avg=950.49 +lavfi.psnr.psnr_avg=18.35 +frame:3 pts:3 pts_time:3 +lavfi.psnr.mse.r=1452.80 +lavfi.psnr.psnr.r=16.51 +lavfi.psnr.mse.g=1001.02 +lavfi.psnr.psnr.g=18.13 +lavfi.psnr.mse.b=557.39 +lavfi.psnr.psnr.b=20.67 +lavfi.psnr.mse_avg=1003.74 +lavfi.psnr.psnr_avg=18.11 +frame:4 pts:4 pts_time:4 +lavfi.psnr.mse.r=1401.25 +lavfi.psnr.psnr.r=16.67 +lavfi.psnr.mse.g=1009.80 +lavfi.psnr.psnr.g=18.09 +lavfi.psnr.mse.b=602.42 +lavfi.psnr.psnr.b=20.33 +lavfi.psnr.mse_avg=1004.49 +lavfi.psnr.psnr_avg=18.11 diff --git a/tests/ref/fate/filter-refcmp-psnr-yuv b/tests/ref/fate/filter-refcmp-psnr-yuv new file mode 100644 index 0000000..942542a --- /dev/null +++ b/tests/ref/fate/filter-refcmp-psnr-yuv @@ -0,0 +1,45 @@ +frame:0 pts:0 pts_time:0 +lavfi.psnr.mse.y=222.06 +lavfi.psnr.psnr.y=24.67 +lavfi.psnr.mse.u=339.38 +lavfi.psnr.psnr.u=22.82 +lavfi.psnr.mse.v=705.41 +lavfi.psnr.psnr.v=19.65 +lavfi.psnr.mse_avg=372.23 +lavfi.psnr.psnr_avg=22.42 +frame:1 pts:1 pts_time:1 +lavfi.psnr.mse.y=236.74 +lavfi.psnr.psnr.y=24.39 +lavfi.psnr.mse.u=416.17 +lavfi.psnr.psnr.u=21.94 +lavfi.psnr.mse.v=704.98 +lavfi.psnr.psnr.v=19.65 +lavfi.psnr.mse_avg=398.66 +lavfi.psnr.psnr_avg=22.12 +frame:2 pts:2 pts_time:2 +lavfi.psnr.mse.y=234.79 +lavfi.psnr.psnr.y=24.42 +lavfi.psnr.mse.u=435.72 +lavfi.psnr.psnr.u=21.74 +lavfi.psnr.mse.v=699.60 +lavfi.psnr.psnr.v=19.68 +lavfi.psnr.mse_avg=401.23 +lavfi.psnr.psnr_avg=22.10 +frame:3 pts:3 pts_time:3 +lavfi.psnr.mse.y=250.88 +lavfi.psnr.psnr.y=24.14 +lavfi.psnr.mse.u=479.73 +lavfi.psnr.psnr.u=21.32 +lavfi.psnr.mse.v=707.55 +lavfi.psnr.psnr.v=19.63 +lavfi.psnr.mse_avg=422.26 +lavfi.psnr.psnr_avg=21.88 +frame:4 pts:4 pts_time:4 +lavfi.psnr.mse.y=241.05 +lavfi.psnr.psnr.y=24.31 +lavfi.psnr.mse.u=505.04 +lavfi.psnr.psnr.u=21.10 +lavfi.psnr.mse.v=716.00 +lavfi.psnr.psnr.v=19.58 +lavfi.psnr.mse_avg=425.79 +lavfi.psnr.psnr_avg=21.84 diff --git a/tests/ref/fate/filter-refcmp-ssim-rgb b/tests/ref/fate/filter-refcmp-ssim-rgb new file mode 100644 index 0000000..4a06eb4 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-ssim-rgb @@ -0,0 +1,30 @@ +frame:0 pts:0 pts_time:0 +lavfi.ssim.R=0.72 +lavfi.ssim.G=0.76 +lavfi.ssim.B=0.89 +lavfi.ssim.All=0.79 +lavfi.ssim.dB=6.74 +frame:1 pts:1 pts_time:1 +lavfi.ssim.R=0.70 +lavfi.ssim.G=0.74 +lavfi.ssim.B=0.85 +lavfi.ssim.All=0.77 +lavfi.ssim.dB=6.31 +frame:2 pts:2 pts_time:2 +lavfi.ssim.R=0.71 +lavfi.ssim.G=0.75 +lavfi.ssim.B=0.84 +lavfi.ssim.All=0.76 +lavfi.ssim.dB=6.29 +frame:3 pts:3 pts_time:3 +lavfi.ssim.R=0.70 +lavfi.ssim.G=0.73 +lavfi.ssim.B=0.83 +lavfi.ssim.All=0.76 +lavfi.ssim.dB=6.11 +frame:4 pts:4 pts_time:4 +lavfi.ssim.R=0.71 +lavfi.ssim.G=0.74 +lavfi.ssim.B=0.80 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=6.05 diff --git a/tests/ref/fate/filter-refcmp-ssim-yuv b/tests/ref/fate/filter-refcmp-ssim-yuv new file mode 100644 index 0000000..0cd46c8 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-ssim-yuv @@ -0,0 +1,30 @@ +frame:0 pts:0 pts_time:0 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.76 +lavfi.ssim.V=0.69 +lavfi.ssim.All=0.76 +lavfi.ssim.dB=6.25 +frame:1 pts:1 pts_time:1 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.73 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=6.08 +frame:2 pts:2 pts_time:2 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.73 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=6.10 +frame:3 pts:3 pts_time:3 +lavfi.ssim.Y=0.79 +lavfi.ssim.U=0.72 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=5.94 +frame:4 pts:4 pts_time:4 +lavfi.ssim.Y=0.80 +lavfi.ssim.U=0.72 +lavfi.ssim.V=0.68 +lavfi.ssim.All=0.75 +lavfi.ssim.dB=5.97 diff --git a/tests/refcmp-metadata.awk b/tests/refcmp-metadata.awk new file mode 100644 index 0000000..c40a11b --- /dev/null +++ b/tests/refcmp-metadata.awk @@ -0,0 +1,61 @@ +# Compare metadata filter output containing float value strings to reference +# output data. Returns the whole reference data if delta of each value is below +# threshold, else returns the whole input data. + +function abs(val) { + return ((val < 0.0) ? -val : val); +} + +function max(val1, val2) { + return ((val1 >= val2) ? val1 : val2); +} + +function is_numeric_str(str) { + return (str ~ /^[+-]?[0-9]*\.?[0-9]+$/); +} + +BEGIN { + FS = "="; + # check for "fuzz" (threshold) program parameter, else use default + if (fuzz <= 0.0) { + fuzz = 0.1; + } + # check for "ref" (reference file) program parameter + if (ref) { + FNR = 0; + while ((getline line < ref) > 0) { + FNR++; + ref_lines[FNR] = line; + if (split(line, fields) == 2 && is_numeric_str(fields[2])) { + ref_keys[FNR] = fields[1]; + ref_vals[FNR] = fields[2] + 0; # convert to number + } + } + close(ref); + } + delta_max = 0; + result = (ref ? 1 : 0); +} + +{ + cmp_lines[FNR] = $0; + if (NF == 2 && is_numeric_str($2) && ref_vals[FNR]) { + val = $2 + 0; # convert to number + delta = abs((val / ref_vals[FNR]) - 1); + delta_max = max(delta_max, delta); + result = result && ($1 == ref_keys[FNR]) && (delta <= fuzz); + } else { + result = result && ($0 == ref_lines[FNR]); + } +} + +END { + if (result) { + for (i in ref_lines) + print ref_lines[i]; + } else { + for (i in cmp_lines) + print cmp_lines[i]; + print "[refcmp] delta_max: " delta_max " >= " fuzz > "/dev/stderr"; + } +}