diff mbox

[FFmpeg-devel] fate: add tests for psnr and ssim filter

Message ID 20170720202144.GG3740@nb4
State Superseded
Headers show

Commit Message

Michael Niedermayer July 20, 2017, 8:21 p.m. UTC 
On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote:
> On 19.07.2017 17:06, Nicolas George wrote:
> >Le primidi 1er thermidor, an CCXXV, Tobias Rapp a écrit :
> >>Indeed they are remainders from previous edits. It seems that stripping any
> >>whitespace within the filter string is enough to ensure that it is passed as
> >>one argument token to ffmpeg. So fixed locally.
> >
> >I suppose you left the quotes, otherwise the semicolon separates
> >commands and the brackets are globbing patterns.
> 
> Indeed, see attached patch for reference (WIP as it still contains
> the CPUFLAGS work-around).
> 
> >>>>+	-f null /dev/null
> >>BTW: Is it OK to redirect output to "/dev/null" here or does this introduce
> >>an unwanted platform dependency (i.e. blocks FATE from running on
> >>MSYS/Windows)?
> >
> >With -f null, you can put anything you want there. I usually put -, but
> >that is just a convenience to be able to change it to "-f fmt - | cmd".
> 
> OK.
> 
> Regards,
> Tobias

>  fate/filter-video.mak           |   16 ++++++++++++++
>  ref/fate/filter-refcmp-psnr-rgb |   45 ++++++++++++++++++++++++++++++++++++++++
>  ref/fate/filter-refcmp-psnr-yuv |   45 ++++++++++++++++++++++++++++++++++++++++
>  ref/fate/filter-refcmp-ssim-rgb |   30 ++++++++++++++++++++++++++
>  ref/fate/filter-refcmp-ssim-yuv |   30 ++++++++++++++++++++++++++
>  5 files changed, 166 insertions(+)
> ba25d086250183b279c65419343103872074dab8  0001-fate-add-tests-for-psnr-and-ssim-filter.patch
> From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001
> From: Tobias Rapp <t.rapp@noa-archive.com>
> Date: Tue, 11 Jul 2017 12:38:06 +0200
> Subject: [PATCH] fate: add tests for psnr and ssim filter
> 
> Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com>

on x86-32:

Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details.
make: *** [fate-filter-refcmp-psnr-rgb] Error 1


[...]

Comments

Tobias Rapp July 21, 2017, 2:39 p.m. UTC | #1 
On 20.07.2017 22:21, Michael Niedermayer wrote:
> On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote:
> [...]
>>  fate/filter-video.mak           |   16 ++++++++++++++
>>  ref/fate/filter-refcmp-psnr-rgb |   45 ++++++++++++++++++++++++++++++++++++++++
>>  ref/fate/filter-refcmp-psnr-yuv |   45 ++++++++++++++++++++++++++++++++++++++++
>>  ref/fate/filter-refcmp-ssim-rgb |   30 ++++++++++++++++++++++++++
>>  ref/fate/filter-refcmp-ssim-yuv |   30 ++++++++++++++++++++++++++
>>  5 files changed, 166 insertions(+)
>> ba25d086250183b279c65419343103872074dab8  0001-fate-add-tests-for-psnr-and-ssim-filter.patch
>> From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001
>> From: Tobias Rapp <t.rapp@noa-archive.com>
>> Date: Tue, 11 Jul 2017 12:38:06 +0200
>> Subject: [PATCH] fate: add tests for psnr and ssim filter
>>
>> Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com>
>
> on x86-32:
>
> --- tests/ref/fate/filter-refcmp-psnr-yuv       2017-07-20 22:05:14.887394601 +0200
> +++ tests/data/fate/filter-refcmp-psnr-yuv      2017-07-20 22:14:51.363406746 +0200
> @@ -3,43 +3,43 @@
>  lavfi.psnr.psnr.y=24.67
>  lavfi.psnr.mse.u=339.38
>  lavfi.psnr.psnr.u=22.82
> -lavfi.psnr.mse.v=705.41
> +lavfi.psnr.mse.v=705.31
>  lavfi.psnr.psnr.v=19.65
> -lavfi.psnr.mse_avg=372.23
> +lavfi.psnr.mse_avg=372.20
>  lavfi.psnr.psnr_avg=22.42
>  frame:1    pts:1       pts_time:1
> -lavfi.psnr.mse.y=236.74
> +lavfi.psnr.mse.y=236.76
>  lavfi.psnr.psnr.y=24.39
>  lavfi.psnr.mse.u=416.17
>  lavfi.psnr.psnr.u=21.94
> -lavfi.psnr.mse.v=704.98
> +lavfi.psnr.mse.v=704.97
>  lavfi.psnr.psnr.v=19.65
>  lavfi.psnr.mse_avg=398.66
>  lavfi.psnr.psnr_avg=22.12
>  frame:2    pts:2       pts_time:2
> -lavfi.psnr.mse.y=234.79
> +lavfi.psnr.mse.y=234.83
>  lavfi.psnr.psnr.y=24.42
>  lavfi.psnr.mse.u=435.72
>  lavfi.psnr.psnr.u=21.74
>  lavfi.psnr.mse.v=699.60
>  lavfi.psnr.psnr.v=19.68
> -lavfi.psnr.mse_avg=401.23
> +lavfi.psnr.mse_avg=401.24
>  lavfi.psnr.psnr_avg=22.10
>  frame:3    pts:3       pts_time:3
> -lavfi.psnr.mse.y=250.88
> +lavfi.psnr.mse.y=250.91
>  lavfi.psnr.psnr.y=24.14
>  lavfi.psnr.mse.u=479.73
>  lavfi.psnr.psnr.u=21.32
> -lavfi.psnr.mse.v=707.55
> +lavfi.psnr.mse.v=707.54
>  lavfi.psnr.psnr.v=19.63
> -lavfi.psnr.mse_avg=422.26
> -lavfi.psnr.psnr_avg=21.88
> +lavfi.psnr.mse_avg=422.27
> +lavfi.psnr.psnr_avg=21.87
>  frame:4    pts:4       pts_time:4
> -lavfi.psnr.mse.y=241.05
> +lavfi.psnr.mse.y=241.04
>  lavfi.psnr.psnr.y=24.31
>  lavfi.psnr.mse.u=505.04
>  lavfi.psnr.psnr.u=21.10
>  lavfi.psnr.mse.v=716.00
>  lavfi.psnr.psnr.v=19.58
> -lavfi.psnr.mse_avg=425.79
> +lavfi.psnr.mse_avg=425.78
>  lavfi.psnr.psnr_avg=21.84
> Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details.
> make: *** [fate-filter-refcmp-psnr-yuv] Error 1
> make: *** Waiting for unfinished jobs....
> --- tests/ref/fate/filter-refcmp-psnr-rgb       2017-07-20 22:05:14.887394601 +0200
> +++ tests/data/fate/filter-refcmp-psnr-rgb      2017-07-20 22:14:51.367406746 +0200
> @@ -1,11 +1,11 @@
>  frame:0    pts:0       pts_time:0
> -lavfi.psnr.mse.r=1381.80
> +lavfi.psnr.mse.r=1381.38
>  lavfi.psnr.psnr.r=16.73
>  lavfi.psnr.mse.g=896.00
>  lavfi.psnr.psnr.g=18.61
>  lavfi.psnr.mse.b=277.38
>  lavfi.psnr.psnr.b=23.70
> -lavfi.psnr.mse_avg=851.73
> +lavfi.psnr.mse_avg=851.59
>  lavfi.psnr.psnr_avg=18.83
>  frame:1    pts:1       pts_time:1
>  lavfi.psnr.mse.r=1380.37
> @@ -17,9 +17,9 @@
>  lavfi.psnr.mse_avg=930.67
>  lavfi.psnr.psnr_avg=18.44
>  frame:2    pts:2       pts_time:2
> -lavfi.psnr.mse.r=1403.20
> +lavfi.psnr.mse.r=1403.21
>  lavfi.psnr.psnr.r=16.66
> -lavfi.psnr.mse.g=954.05
> +lavfi.psnr.mse.g=954.04
>  lavfi.psnr.psnr.g=18.34
>  lavfi.psnr.mse.b=494.22
>  lavfi.psnr.psnr.b=21.19
> Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details.
> make: *** [fate-filter-refcmp-psnr-rgb] Error 1
>

So I assume the lesson to be learned for me should be that it is not 
safe to compare string-ified floats and a sane test should do something 
like "abs(ref_val - curr_val) < eps" for each metadata entry?

Regards,
Tobias
Michael Niedermayer July 21, 2017, 11:19 p.m. UTC | #2 
On Fri, Jul 21, 2017 at 04:39:37PM +0200, Tobias Rapp wrote:
> On 20.07.2017 22:21, Michael Niedermayer wrote:
> >On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote:
> >[...]
> >> fate/filter-video.mak           |   16 ++++++++++++++
> >> ref/fate/filter-refcmp-psnr-rgb |   45 ++++++++++++++++++++++++++++++++++++++++
> >> ref/fate/filter-refcmp-psnr-yuv |   45 ++++++++++++++++++++++++++++++++++++++++
> >> ref/fate/filter-refcmp-ssim-rgb |   30 ++++++++++++++++++++++++++
> >> ref/fate/filter-refcmp-ssim-yuv |   30 ++++++++++++++++++++++++++
> >> 5 files changed, 166 insertions(+)
> >>ba25d086250183b279c65419343103872074dab8  0001-fate-add-tests-for-psnr-and-ssim-filter.patch
> >>From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001
> >>From: Tobias Rapp <t.rapp@noa-archive.com>
> >>Date: Tue, 11 Jul 2017 12:38:06 +0200
> >>Subject: [PATCH] fate: add tests for psnr and ssim filter
> >>
> >>Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com>
> >
> >on x86-32:
> >
> >--- tests/ref/fate/filter-refcmp-psnr-yuv       2017-07-20 22:05:14.887394601 +0200
> >+++ tests/data/fate/filter-refcmp-psnr-yuv      2017-07-20 22:14:51.363406746 +0200
> >@@ -3,43 +3,43 @@
> > lavfi.psnr.psnr.y=24.67
> > lavfi.psnr.mse.u=339.38
> > lavfi.psnr.psnr.u=22.82
> >-lavfi.psnr.mse.v=705.41
> >+lavfi.psnr.mse.v=705.31
> > lavfi.psnr.psnr.v=19.65
> >-lavfi.psnr.mse_avg=372.23
> >+lavfi.psnr.mse_avg=372.20
> > lavfi.psnr.psnr_avg=22.42
> > frame:1    pts:1       pts_time:1
> >-lavfi.psnr.mse.y=236.74
> >+lavfi.psnr.mse.y=236.76
> > lavfi.psnr.psnr.y=24.39
> > lavfi.psnr.mse.u=416.17
> > lavfi.psnr.psnr.u=21.94
> >-lavfi.psnr.mse.v=704.98
> >+lavfi.psnr.mse.v=704.97
> > lavfi.psnr.psnr.v=19.65
> > lavfi.psnr.mse_avg=398.66
> > lavfi.psnr.psnr_avg=22.12
> > frame:2    pts:2       pts_time:2
> >-lavfi.psnr.mse.y=234.79
> >+lavfi.psnr.mse.y=234.83
> > lavfi.psnr.psnr.y=24.42
> > lavfi.psnr.mse.u=435.72
> > lavfi.psnr.psnr.u=21.74
> > lavfi.psnr.mse.v=699.60
> > lavfi.psnr.psnr.v=19.68
> >-lavfi.psnr.mse_avg=401.23
> >+lavfi.psnr.mse_avg=401.24
> > lavfi.psnr.psnr_avg=22.10
> > frame:3    pts:3       pts_time:3
> >-lavfi.psnr.mse.y=250.88
> >+lavfi.psnr.mse.y=250.91
> > lavfi.psnr.psnr.y=24.14
> > lavfi.psnr.mse.u=479.73
> > lavfi.psnr.psnr.u=21.32
> >-lavfi.psnr.mse.v=707.55
> >+lavfi.psnr.mse.v=707.54
> > lavfi.psnr.psnr.v=19.63
> >-lavfi.psnr.mse_avg=422.26
> >-lavfi.psnr.psnr_avg=21.88
> >+lavfi.psnr.mse_avg=422.27
> >+lavfi.psnr.psnr_avg=21.87
> > frame:4    pts:4       pts_time:4
> >-lavfi.psnr.mse.y=241.05
> >+lavfi.psnr.mse.y=241.04
> > lavfi.psnr.psnr.y=24.31
> > lavfi.psnr.mse.u=505.04
> > lavfi.psnr.psnr.u=21.10
> > lavfi.psnr.mse.v=716.00
> > lavfi.psnr.psnr.v=19.58
> >-lavfi.psnr.mse_avg=425.79
> >+lavfi.psnr.mse_avg=425.78
> > lavfi.psnr.psnr_avg=21.84
> >Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details.
> >make: *** [fate-filter-refcmp-psnr-yuv] Error 1
> >make: *** Waiting for unfinished jobs....
> >--- tests/ref/fate/filter-refcmp-psnr-rgb       2017-07-20 22:05:14.887394601 +0200
> >+++ tests/data/fate/filter-refcmp-psnr-rgb      2017-07-20 22:14:51.367406746 +0200
> >@@ -1,11 +1,11 @@
> > frame:0    pts:0       pts_time:0
> >-lavfi.psnr.mse.r=1381.80
> >+lavfi.psnr.mse.r=1381.38
> > lavfi.psnr.psnr.r=16.73
> > lavfi.psnr.mse.g=896.00
> > lavfi.psnr.psnr.g=18.61
> > lavfi.psnr.mse.b=277.38
> > lavfi.psnr.psnr.b=23.70
> >-lavfi.psnr.mse_avg=851.73
> >+lavfi.psnr.mse_avg=851.59
> > lavfi.psnr.psnr_avg=18.83
> > frame:1    pts:1       pts_time:1
> > lavfi.psnr.mse.r=1380.37
> >@@ -17,9 +17,9 @@
> > lavfi.psnr.mse_avg=930.67
> > lavfi.psnr.psnr_avg=18.44
> > frame:2    pts:2       pts_time:2
> >-lavfi.psnr.mse.r=1403.20
> >+lavfi.psnr.mse.r=1403.21
> > lavfi.psnr.psnr.r=16.66
> >-lavfi.psnr.mse.g=954.05
> >+lavfi.psnr.mse.g=954.04
> > lavfi.psnr.psnr.g=18.34
> > lavfi.psnr.mse.b=494.22
> > lavfi.psnr.psnr.b=21.19
> >Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details.
> >make: *** [fate-filter-refcmp-psnr-rgb] Error 1
> >
> 
> So I assume the lesson to be learned for me should be that it is not
> safe to compare string-ified floats and a sane test should do
> something like "abs(ref_val - curr_val) < eps" for each metadata
> entry?

we might be able to get away with strings and lower precission.
but yes otherwise with inexact code like floats checking the difference
would likely be needed

[...]
Nicolas George July 23, 2017, 11:30 a.m. UTC | #3 
Le quartidi 4 thermidor, an CCXXV, Michael Niedermayer a écrit :
> we might be able to get away with strings and lower precission.

We might be lucky like that, but we cannot expect it to work always: the
value always has a small chance to be too near the cutoff value between
two rounded numbers.

Also, decreasing the precision increases the risk of missing a
regression.

I just wrote the attached script, it may or may not be of use.

Regards,
diff mbox

Patch

--- tests/ref/fate/filter-refcmp-psnr-yuv       2017-07-20 22:05:14.887394601 +0200
+++ tests/data/fate/filter-refcmp-psnr-yuv      2017-07-20 22:14:51.363406746 +0200
@@ -3,43 +3,43 @@ 
 lavfi.psnr.psnr.y=24.67
 lavfi.psnr.mse.u=339.38
 lavfi.psnr.psnr.u=22.82
-lavfi.psnr.mse.v=705.41
+lavfi.psnr.mse.v=705.31
 lavfi.psnr.psnr.v=19.65
-lavfi.psnr.mse_avg=372.23
+lavfi.psnr.mse_avg=372.20
 lavfi.psnr.psnr_avg=22.42
 frame:1    pts:1       pts_time:1
-lavfi.psnr.mse.y=236.74
+lavfi.psnr.mse.y=236.76
 lavfi.psnr.psnr.y=24.39
 lavfi.psnr.mse.u=416.17
 lavfi.psnr.psnr.u=21.94
-lavfi.psnr.mse.v=704.98
+lavfi.psnr.mse.v=704.97
 lavfi.psnr.psnr.v=19.65
 lavfi.psnr.mse_avg=398.66
 lavfi.psnr.psnr_avg=22.12
 frame:2    pts:2       pts_time:2
-lavfi.psnr.mse.y=234.79
+lavfi.psnr.mse.y=234.83
 lavfi.psnr.psnr.y=24.42
 lavfi.psnr.mse.u=435.72
 lavfi.psnr.psnr.u=21.74
 lavfi.psnr.mse.v=699.60
 lavfi.psnr.psnr.v=19.68
-lavfi.psnr.mse_avg=401.23
+lavfi.psnr.mse_avg=401.24
 lavfi.psnr.psnr_avg=22.10
 frame:3    pts:3       pts_time:3
-lavfi.psnr.mse.y=250.88
+lavfi.psnr.mse.y=250.91
 lavfi.psnr.psnr.y=24.14
 lavfi.psnr.mse.u=479.73
 lavfi.psnr.psnr.u=21.32
-lavfi.psnr.mse.v=707.55
+lavfi.psnr.mse.v=707.54
 lavfi.psnr.psnr.v=19.63
-lavfi.psnr.mse_avg=422.26
-lavfi.psnr.psnr_avg=21.88
+lavfi.psnr.mse_avg=422.27
+lavfi.psnr.psnr_avg=21.87
 frame:4    pts:4       pts_time:4
-lavfi.psnr.mse.y=241.05
+lavfi.psnr.mse.y=241.04
 lavfi.psnr.psnr.y=24.31
 lavfi.psnr.mse.u=505.04
 lavfi.psnr.psnr.u=21.10
 lavfi.psnr.mse.v=716.00
 lavfi.psnr.psnr.v=19.58
-lavfi.psnr.mse_avg=425.79
+lavfi.psnr.mse_avg=425.78
 lavfi.psnr.psnr_avg=21.84
Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details.
make: *** [fate-filter-refcmp-psnr-yuv] Error 1
make: *** Waiting for unfinished jobs....
--- tests/ref/fate/filter-refcmp-psnr-rgb       2017-07-20 22:05:14.887394601 +0200
+++ tests/data/fate/filter-refcmp-psnr-rgb      2017-07-20 22:14:51.367406746 +0200
@@ -1,11 +1,11 @@ 
 frame:0    pts:0       pts_time:0
-lavfi.psnr.mse.r=1381.80
+lavfi.psnr.mse.r=1381.38
 lavfi.psnr.psnr.r=16.73
 lavfi.psnr.mse.g=896.00
 lavfi.psnr.psnr.g=18.61
 lavfi.psnr.mse.b=277.38
 lavfi.psnr.psnr.b=23.70
-lavfi.psnr.mse_avg=851.73
+lavfi.psnr.mse_avg=851.59
 lavfi.psnr.psnr_avg=18.83
 frame:1    pts:1       pts_time:1
 lavfi.psnr.mse.r=1380.37
@@ -17,9 +17,9 @@ 
 lavfi.psnr.mse_avg=930.67
 lavfi.psnr.psnr_avg=18.44
 frame:2    pts:2       pts_time:2
-lavfi.psnr.mse.r=1403.20
+lavfi.psnr.mse.r=1403.21
 lavfi.psnr.psnr.r=16.66
-lavfi.psnr.mse.g=954.05
+lavfi.psnr.mse.g=954.04
 lavfi.psnr.psnr.g=18.34
 lavfi.psnr.mse.b=494.22
 lavfi.psnr.psnr.b=21.19