diff mbox

[FFmpeg-devel] fate: allow temp files for passed test to be kept

Message ID 714fc645-6e4b-7e68-44eb-73c78187fcfd@gmail.com
State Accepted
Commit 0bd48ab2d9e463b75ef52c0eaa0cc00c4c385cce
Headers show

Commit Message

Gyan July 6, 2018, 9:35 a.m. UTC 
From 18351747e460d52939d54bdf20364e4156852034 Mon Sep 17 00:00:00 2001
From: Gyan Doshi <ffmpeg@gyani.pro>
Date: Fri, 6 Jul 2018 14:53:33 +0530
Subject: [PATCH] fate: allow temp files for passed test to be kept

Set make variable KEEP to non-zero value to preserve temp files
when a test has passed.

Helpful in diagnosing failed tests when test outfile is some type of
single hash and does not reveal differences in processed output.
---
 doc/fate.texi     | 5 +++++
 tests/Makefile    | 2 +-
 tests/fate-run.sh | 5 ++++-
 3 files changed, 10 insertions(+), 2 deletions(-)

Comments

Gyan July 8, 2018, 5:57 p.m. UTC | #1 
Ping.
Gyan July 9, 2018, 4:36 a.m. UTC | #2 
On 08-07-2018 11:27 PM, Gyan Doshi wrote:
> 
> Ping.

Will push tonight if no objections.
Gyan July 10, 2018, 4:44 a.m. UTC | #3 
On 09-07-2018 10:06 AM, Gyan Doshi wrote:
> 
> Will push tonight if no objections.

Pushed as 0bd48ab2d9e463b75ef52c0eaa0cc00c4c385cce
Carl Eugen Hoyos July 10, 2018, 12:10 p.m. UTC | #4 
2018-07-06 11:35 GMT+02:00, Gyan Doshi <gyandoshi@gmail.com>:

> +@item KEEP
> +Set to @samp{1} to keep temp files generated by fate test(s) when test is successful.
> +Default is @samp{0}, which removes these files. Files are always kept when a test
> +fails.

Sorry for the late comment:
Shouldn't this be -1 for keep if fail, 0 for always delete, 1 for keep?

Carl Eugen
Gyan July 10, 2018, 12:46 p.m. UTC | #5 
On 10-07-2018 05:40 PM, Carl Eugen Hoyos wrote:

> Shouldn't this be -1 for keep if fail, 0 for always delete, 1 for keep?

$keep is checked only when $err = 0 i.e.

a) passed test or
b) failed test and GEN != no

In the 2nd case, since user has decided to overwrite existing ref, I 
assume the action is post-diagnosis and the failure doesn't signal a
regression.

Regards,
Gyan
diff mbox

Patch

diff --git a/doc/fate.texi b/doc/fate.texi
index c9c0d3aea9..a352994230 100644
--- a/doc/fate.texi
+++ b/doc/fate.texi
@@ -222,6 +222,11 @@  Set to @samp{1} to generate the missing or mismatched references.
 Specify which hardware acceleration to use while running regression tests,
 by default @samp{none} is used.
 
+@item KEEP
+Set to @samp{1} to keep temp files generated by fate test(s) when test is successful.
+Default is @samp{0}, which removes these files. Files are always kept when a test
+fails.
+
 @end table
 
 @section Examples
diff --git a/tests/Makefile b/tests/Makefile
index e564279b0a..24680b8150 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -225,7 +225,7 @@  FATE += $(FATE_HW-yes)
 $(FATE) $(FATE_TESTS-no): export PROGSUF = $(PROGSSUF)
 $(FATE) $(FATE_TESTS-no): $(FATE_UTILS:%=tests/%$(HOSTEXESUF))
 	@echo "TEST    $(@:fate-%=%)"
-	$(Q)$(SRC_PATH)/tests/fate-run.sh $@ "$(TARGET_SAMPLES)" "$(TARGET_EXEC)" "$(TARGET_PATH)" '$(CMD)' '$(CMP)' '$(REF)' '$(FUZZ)' '$(THREADS)' '$(THREAD_TYPE)' '$(CPUFLAGS)' '$(CMP_SHIFT)' '$(CMP_TARGET)' '$(SIZE_TOLERANCE)' '$(CMP_UNIT)' '$(GEN)' '$(HWACCEL)' '$(REPORT)'
+	$(Q)$(SRC_PATH)/tests/fate-run.sh $@ "$(TARGET_SAMPLES)" "$(TARGET_EXEC)" "$(TARGET_PATH)" '$(CMD)' '$(CMP)' '$(REF)' '$(FUZZ)' '$(THREADS)' '$(THREAD_TYPE)' '$(CPUFLAGS)' '$(CMP_SHIFT)' '$(CMP_TARGET)' '$(SIZE_TOLERANCE)' '$(CMP_UNIT)' '$(GEN)' '$(HWACCEL)' '$(REPORT)' '$(KEEP)'
 
 fate-list:
 	@printf '%s\n' $(sort $(FATE))
diff --git a/tests/fate-run.sh b/tests/fate-run.sh
index 457761c152..aece90a01d 100755
--- a/tests/fate-run.sh
+++ b/tests/fate-run.sh
@@ -25,6 +25,7 @@  cmp_unit=${15:-2}
 gen=${16:-no}
 hwaccel=${17:-none}
 report_type=${18:-standard}
+keep=${19:-0}
 
 outdir="tests/data/fate"
 outfile="${outdir}/${test}"
@@ -428,7 +429,9 @@  if test $err != 0 && test $gen != "no" ; then
 fi
 
 if test $err = 0; then
-    rm -f $outfile $errfile $cmpfile $cleanfiles
+    if test $keep = 0; then
+        rm -f $outfile $errfile $cmpfile $cleanfiles
+    fi
 elif test $gen = "no"; then
     echo "Test $test failed. Look at $errfile for details."
     test "${V:-0}" -gt 0 && cat $errfile