diff mbox

[FFmpeg-devel,v3] fate: add tests for psnr and ssim filter

Message ID 20170919233843.GG7094@nb4
State Not Applicable
Headers show

Commit Message

Michael Niedermayer Sept. 19, 2017, 11:38 p.m. UTC 
On Tue, Sep 19, 2017 at 09:48:45AM +0200, Tobias Rapp wrote:
> Metadata filter output is passed through an Awk script comparing floats
> against reference values with specified "fuzz" tolerance to account for
> architectural differences (e.g. x86-32 vs. x86-64).
> 
> Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com>
> ---
> tested on Linux x86-32/64 and Mips (Qemu)
> 
> v3:
>  - avoid precision loss due to rounding of float values
> 
> v2:
>  - removed CPUFLAGS work-around for ssim filter issue
>  - added metadata float value post-processing script
> 
>  tests/fate-run.sh                     |  9 ++++++
>  tests/fate/filter-video.mak           | 14 ++++++++
>  tests/ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++
>  tests/ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++
>  tests/ref/fate/filter-refcmp-ssim-rgb | 30 +++++++++++++++++
>  tests/ref/fate/filter-refcmp-ssim-yuv | 30 +++++++++++++++++
>  tests/refcmp-metadata.awk             | 61 +++++++++++++++++++++++++++++++++++
>  7 files changed, 234 insertions(+)
>  create mode 100644 tests/ref/fate/filter-refcmp-psnr-rgb
>  create mode 100644 tests/ref/fate/filter-refcmp-psnr-yuv
>  create mode 100644 tests/ref/fate/filter-refcmp-ssim-rgb
>  create mode 100644 tests/ref/fate/filter-refcmp-ssim-yuv
>  create mode 100644 tests/refcmp-metadata.awk

This seems not to work (normal x86-64 linux) but maybe ive done something silly

Test filter-refcmp-ssim-yuv failed. Look at tests/data/fate/filter-refcmp-ssim-yuv.err for details.
make: *** [fate-filter-refcmp-ssim-yuv] Error 1

[...]

Comments

Tobias Rapp Sept. 20, 2017, 7:29 a.m. UTC | #1 
On 20.09.2017 01:38, Michael Niedermayer wrote:
> On Tue, Sep 19, 2017 at 09:48:45AM +0200, Tobias Rapp wrote:
>> Metadata filter output is passed through an Awk script comparing floats
>> against reference values with specified "fuzz" tolerance to account for
>> architectural differences (e.g. x86-32 vs. x86-64).
>>
>> Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com>
>> ---
>> tested on Linux x86-32/64 and Mips (Qemu)
>>
>> v3:
>>   - avoid precision loss due to rounding of float values
>>
>> v2:
>>   - removed CPUFLAGS work-around for ssim filter issue
>>   - added metadata float value post-processing script
>>
>>   tests/fate-run.sh                     |  9 ++++++
>>   tests/fate/filter-video.mak           | 14 ++++++++
>>   tests/ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++
>>   tests/ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++
>>   tests/ref/fate/filter-refcmp-ssim-rgb | 30 +++++++++++++++++
>>   tests/ref/fate/filter-refcmp-ssim-yuv | 30 +++++++++++++++++
>>   tests/refcmp-metadata.awk             | 61 +++++++++++++++++++++++++++++++++++
>>   7 files changed, 234 insertions(+)
>>   create mode 100644 tests/ref/fate/filter-refcmp-psnr-rgb
>>   create mode 100644 tests/ref/fate/filter-refcmp-psnr-yuv
>>   create mode 100644 tests/ref/fate/filter-refcmp-ssim-rgb
>>   create mode 100644 tests/ref/fate/filter-refcmp-ssim-yuv
>>   create mode 100644 tests/refcmp-metadata.awk
> 
> This seems not to work (normal x86-64 linux) but maybe ive done something silly
> 
> --- ./tests/ref/fate/filter-refcmp-psnr-rgb     2017-09-20 00:58:04.063337500 +0200
> +++ tests/data/fate/filter-refcmp-psnr-rgb      2017-09-20 00:59:26.399339234 +0200
> @@ -1,12 +1,3 @@
> -frame:0    pts:0       pts_time:0
> -lavfi.psnr.mse.r=1381.80
> -lavfi.psnr.psnr.r=16.73
> -lavfi.psnr.mse.g=896.00
> -lavfi.psnr.psnr.g=18.61
> -lavfi.psnr.mse.b=277.38
> -lavfi.psnr.psnr.b=23.70
> -lavfi.psnr.mse_avg=851.73
> -lavfi.psnr.psnr_avg=18.83
>   frame:1    pts:1       pts_time:1
>   lavfi.psnr.mse.r=1380.37
>   lavfi.psnr.psnr.r=16.73
> [...]

It seems that my assumption that iterating a for-in loop with numerical 
indexes would walk the array in linear order does not apply to all 
versions of Awk. I'm testing with GNU Awk 4.1.3, what version of Awk is 
installed on your computer?

Will rework the loop and post a patch update.

Regards,
Tobias
Michael Niedermayer Sept. 20, 2017, 11:23 p.m. UTC | #2 
On Wed, Sep 20, 2017 at 09:29:34AM +0200, Tobias Rapp wrote:
> On 20.09.2017 01:38, Michael Niedermayer wrote:
> >On Tue, Sep 19, 2017 at 09:48:45AM +0200, Tobias Rapp wrote:
> >>Metadata filter output is passed through an Awk script comparing floats
> >>against reference values with specified "fuzz" tolerance to account for
> >>architectural differences (e.g. x86-32 vs. x86-64).
> >>
> >>Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com>
> >>---
> >>tested on Linux x86-32/64 and Mips (Qemu)
> >>
> >>v3:
> >>  - avoid precision loss due to rounding of float values
> >>
> >>v2:
> >>  - removed CPUFLAGS work-around for ssim filter issue
> >>  - added metadata float value post-processing script
> >>
> >>  tests/fate-run.sh                     |  9 ++++++
> >>  tests/fate/filter-video.mak           | 14 ++++++++
> >>  tests/ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++
> >>  tests/ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++
> >>  tests/ref/fate/filter-refcmp-ssim-rgb | 30 +++++++++++++++++
> >>  tests/ref/fate/filter-refcmp-ssim-yuv | 30 +++++++++++++++++
> >>  tests/refcmp-metadata.awk             | 61 +++++++++++++++++++++++++++++++++++
> >>  7 files changed, 234 insertions(+)
> >>  create mode 100644 tests/ref/fate/filter-refcmp-psnr-rgb
> >>  create mode 100644 tests/ref/fate/filter-refcmp-psnr-yuv
> >>  create mode 100644 tests/ref/fate/filter-refcmp-ssim-rgb
> >>  create mode 100644 tests/ref/fate/filter-refcmp-ssim-yuv
> >>  create mode 100644 tests/refcmp-metadata.awk
> >
> >This seems not to work (normal x86-64 linux) but maybe ive done something silly
> >
> >--- ./tests/ref/fate/filter-refcmp-psnr-rgb     2017-09-20 00:58:04.063337500 +0200
> >+++ tests/data/fate/filter-refcmp-psnr-rgb      2017-09-20 00:59:26.399339234 +0200
> >@@ -1,12 +1,3 @@
> >-frame:0    pts:0       pts_time:0
> >-lavfi.psnr.mse.r=1381.80
> >-lavfi.psnr.psnr.r=16.73
> >-lavfi.psnr.mse.g=896.00
> >-lavfi.psnr.psnr.g=18.61
> >-lavfi.psnr.mse.b=277.38
> >-lavfi.psnr.psnr.b=23.70
> >-lavfi.psnr.mse_avg=851.73
> >-lavfi.psnr.psnr_avg=18.83
> >  frame:1    pts:1       pts_time:1
> >  lavfi.psnr.mse.r=1380.37
> >  lavfi.psnr.psnr.r=16.73
> >[...]
> 
> It seems that my assumption that iterating a for-in loop with
> numerical indexes would walk the array in linear order does not
> apply to all versions of Awk. I'm testing with GNU Awk 4.1.3, what
> version of Awk is installed on your computer?

there are a few awk versions installed but

awk --version
GNU Awk 4.0.1

[...]
diff mbox

Patch

--- ./tests/ref/fate/filter-refcmp-psnr-rgb     2017-09-20 00:58:04.063337500 +0200
+++ tests/data/fate/filter-refcmp-psnr-rgb      2017-09-20 00:59:26.399339234 +0200
@@ -1,12 +1,3 @@ 
-frame:0    pts:0       pts_time:0
-lavfi.psnr.mse.r=1381.80
-lavfi.psnr.psnr.r=16.73
-lavfi.psnr.mse.g=896.00
-lavfi.psnr.psnr.g=18.61
-lavfi.psnr.mse.b=277.38
-lavfi.psnr.psnr.b=23.70
-lavfi.psnr.mse_avg=851.73
-lavfi.psnr.psnr_avg=18.83
 frame:1    pts:1       pts_time:1
 lavfi.psnr.mse.r=1380.37
 lavfi.psnr.psnr.r=16.73
@@ -14,22 +5,12 @@ 
 lavfi.psnr.psnr.g=18.24
 lavfi.psnr.mse.b=435.72
 lavfi.psnr.psnr.b=21.74
-lavfi.psnr.mse_avg=930.67
-lavfi.psnr.psnr_avg=18.44
-frame:2    pts:2       pts_time:2
-lavfi.psnr.mse.r=1403.20
-lavfi.psnr.psnr.r=16.66
-lavfi.psnr.mse.g=954.05
-lavfi.psnr.psnr.g=18.34
-lavfi.psnr.mse.b=494.22
-lavfi.psnr.psnr.b=21.19
-lavfi.psnr.mse_avg=950.49
-lavfi.psnr.psnr_avg=18.35
-frame:3    pts:3       pts_time:3
-lavfi.psnr.mse.r=1452.80
 lavfi.psnr.psnr.r=16.51
+lavfi.psnr.mse_avg=930.67
 lavfi.psnr.mse.g=1001.02
+lavfi.psnr.psnr_avg=18.44
 lavfi.psnr.psnr.g=18.13
+frame:2    pts:2       pts_time:2
 lavfi.psnr.mse.b=557.39
 lavfi.psnr.psnr.b=20.67
 lavfi.psnr.mse_avg=1003.74
@@ -37,9 +18,28 @@ 
 frame:4    pts:4       pts_time:4
 lavfi.psnr.mse.r=1401.25
 lavfi.psnr.psnr.r=16.67
+frame:0    pts:0       pts_time:0
+lavfi.psnr.mse.r=1381.80
+lavfi.psnr.psnr.r=16.73
+lavfi.psnr.mse.g=896.00
+lavfi.psnr.psnr.g=18.61
+lavfi.psnr.mse.b=277.38
+lavfi.psnr.psnr.b=23.70
+lavfi.psnr.mse_avg=851.73
+lavfi.psnr.psnr_avg=18.83
+lavfi.psnr.mse.r=1403.20
+lavfi.psnr.psnr.r=16.66
+lavfi.psnr.mse.g=954.05
+lavfi.psnr.psnr.g=18.34
+lavfi.psnr.mse.b=494.22
+lavfi.psnr.psnr.b=21.19
+lavfi.psnr.mse_avg=950.49
 lavfi.psnr.mse.g=1009.80
+lavfi.psnr.psnr_avg=18.35
 lavfi.psnr.psnr.g=18.09
+frame:3    pts:3       pts_time:3
 lavfi.psnr.mse.b=602.42
+lavfi.psnr.mse.r=1452.80
 lavfi.psnr.psnr.b=20.33
 lavfi.psnr.mse_avg=1004.49
 lavfi.psnr.psnr_avg=18.11
Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details.
make: *** [fate-filter-refcmp-psnr-rgb] Error 1
TEST    fitsdec-bitpix-32
--- ./tests/ref/fate/filter-refcmp-ssim-rgb     2017-09-20 00:58:04.063337500 +0200
+++ tests/data/fate/filter-refcmp-ssim-rgb      2017-09-20 00:59:26.415339235 +0200
@@ -1,30 +1,30 @@ 
-frame:0    pts:0       pts_time:0
-lavfi.ssim.R=0.72
-lavfi.ssim.G=0.76
+lavfi.ssim.R=0.71
+lavfi.ssim.All=0.76
 lavfi.ssim.B=0.89
+lavfi.ssim.G=0.74
+lavfi.ssim.dB=6.29
 lavfi.ssim.All=0.79
+lavfi.ssim.B=0.80
+frame:3    pts:3       pts_time:3
 lavfi.ssim.dB=6.74
+lavfi.ssim.All=0.75
 frame:1    pts:1       pts_time:1
 lavfi.ssim.R=0.70
 lavfi.ssim.G=0.74
 lavfi.ssim.B=0.85
-lavfi.ssim.All=0.77
-lavfi.ssim.dB=6.31
-frame:2    pts:2       pts_time:2
-lavfi.ssim.R=0.71
-lavfi.ssim.G=0.75
-lavfi.ssim.B=0.84
-lavfi.ssim.All=0.76
-lavfi.ssim.dB=6.29
-frame:3    pts:3       pts_time:3
 lavfi.ssim.R=0.70
+lavfi.ssim.All=0.77
+lavfi.ssim.dB=6.05
 lavfi.ssim.G=0.73
+lavfi.ssim.dB=6.31
 lavfi.ssim.B=0.83
+frame:2    pts:2       pts_time:2
 lavfi.ssim.All=0.76
+lavfi.ssim.R=0.71
+frame:0    pts:0       pts_time:0
 lavfi.ssim.dB=6.11
+lavfi.ssim.G=0.75
+lavfi.ssim.R=0.72
 frame:4    pts:4       pts_time:4
-lavfi.ssim.R=0.71
-lavfi.ssim.G=0.74
-lavfi.ssim.B=0.80
-lavfi.ssim.All=0.75
-lavfi.ssim.dB=6.05
+lavfi.ssim.B=0.84
+lavfi.ssim.G=0.76
Test filter-refcmp-ssim-rgb failed. Look at tests/data/fate/filter-refcmp-ssim-rgb.err for details.
make: *** [fate-filter-refcmp-ssim-rgb] Error 1
TEST    fitsdec-bitpix-64
--- ./tests/ref/fate/filter-refcmp-psnr-yuv     2017-09-20 00:58:04.063337500 +0200
+++ tests/data/fate/filter-refcmp-psnr-yuv      2017-09-20 00:59:26.435339235 +0200
@@ -1,12 +1,3 @@ 
-frame:0    pts:0       pts_time:0
-lavfi.psnr.mse.y=222.06
-lavfi.psnr.psnr.y=24.67
-lavfi.psnr.mse.u=339.38
-lavfi.psnr.psnr.u=22.82
-lavfi.psnr.mse.v=705.41
-lavfi.psnr.psnr.v=19.65
-lavfi.psnr.mse_avg=372.23
-lavfi.psnr.psnr_avg=22.42
 frame:1    pts:1       pts_time:1
 lavfi.psnr.mse.y=236.74
 lavfi.psnr.psnr.y=24.39
@@ -14,22 +5,12 @@ 
 lavfi.psnr.psnr.u=21.94
 lavfi.psnr.mse.v=704.98
 lavfi.psnr.psnr.v=19.65
-lavfi.psnr.mse_avg=398.66
-lavfi.psnr.psnr_avg=22.12
-frame:2    pts:2       pts_time:2
-lavfi.psnr.mse.y=234.79
-lavfi.psnr.psnr.y=24.42
-lavfi.psnr.mse.u=435.72
-lavfi.psnr.psnr.u=21.74
-lavfi.psnr.mse.v=699.60
-lavfi.psnr.psnr.v=19.68
-lavfi.psnr.mse_avg=401.23
-lavfi.psnr.psnr_avg=22.10
-frame:3    pts:3       pts_time:3
-lavfi.psnr.mse.y=250.88
 lavfi.psnr.psnr.y=24.14
+lavfi.psnr.mse_avg=398.66
 lavfi.psnr.mse.u=479.73
+lavfi.psnr.psnr_avg=22.12
 lavfi.psnr.psnr.u=21.32
+frame:2    pts:2       pts_time:2
 lavfi.psnr.mse.v=707.55
 lavfi.psnr.psnr.v=19.63
 lavfi.psnr.mse_avg=422.26
@@ -37,9 +18,28 @@ 
 frame:4    pts:4       pts_time:4
 lavfi.psnr.mse.y=241.05
 lavfi.psnr.psnr.y=24.31
+frame:0    pts:0       pts_time:0
+lavfi.psnr.mse.y=222.06
+lavfi.psnr.psnr.y=24.67
+lavfi.psnr.mse.u=339.38
+lavfi.psnr.psnr.u=22.82
+lavfi.psnr.mse.v=705.41
+lavfi.psnr.psnr.v=19.65
+lavfi.psnr.mse_avg=372.23
+lavfi.psnr.psnr_avg=22.42
+lavfi.psnr.mse.y=234.79
+lavfi.psnr.psnr.y=24.42
+lavfi.psnr.mse.u=435.72
+lavfi.psnr.psnr.u=21.74
+lavfi.psnr.mse.v=699.60
+lavfi.psnr.psnr.v=19.68
+lavfi.psnr.mse_avg=401.23
 lavfi.psnr.mse.u=505.04
+lavfi.psnr.psnr_avg=22.10
 lavfi.psnr.psnr.u=21.10
+frame:3    pts:3       pts_time:3
 lavfi.psnr.mse.v=716.00
+lavfi.psnr.mse.y=250.88
 lavfi.psnr.psnr.v=19.58
 lavfi.psnr.mse_avg=425.79
 lavfi.psnr.psnr_avg=21.84
Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details.
make: *** [fate-filter-refcmp-psnr-yuv] Error 1
TEST    fitsdec-multi
GEN     tests/data/lena-gray.fits
GEN     tests/data/lena-gbrp.fits
--- ./tests/ref/fate/filter-refcmp-ssim-yuv     2017-09-20 00:58:04.063337500 +0200
+++ tests/data/fate/filter-refcmp-ssim-yuv      2017-09-20 00:59:26.491339236 +0200
@@ -1,30 +1,30 @@ 
-frame:0    pts:0       pts_time:0
 lavfi.ssim.Y=0.80
-lavfi.ssim.U=0.76
+lavfi.ssim.All=0.75
 lavfi.ssim.V=0.69
+lavfi.ssim.U=0.72
+lavfi.ssim.dB=6.10
 lavfi.ssim.All=0.76
-lavfi.ssim.dB=6.25
-frame:1    pts:1       pts_time:1
-lavfi.ssim.Y=0.80
-lavfi.ssim.U=0.73
 lavfi.ssim.V=0.68
+frame:3    pts:3       pts_time:3
+lavfi.ssim.dB=6.25
 lavfi.ssim.All=0.75
-lavfi.ssim.dB=6.08
-frame:2    pts:2       pts_time:2
+frame:1    pts:1       pts_time:1
 lavfi.ssim.Y=0.80
 lavfi.ssim.U=0.73
 lavfi.ssim.V=0.68
-lavfi.ssim.All=0.75
-lavfi.ssim.dB=6.10
-frame:3    pts:3       pts_time:3
 lavfi.ssim.Y=0.79
+lavfi.ssim.All=0.75
+lavfi.ssim.dB=5.97
 lavfi.ssim.U=0.72
+lavfi.ssim.dB=6.08
 lavfi.ssim.V=0.68
+frame:2    pts:2       pts_time:2
 lavfi.ssim.All=0.75
+lavfi.ssim.Y=0.80
+frame:0    pts:0       pts_time:0
 lavfi.ssim.dB=5.94
-frame:4    pts:4       pts_time:4
+lavfi.ssim.U=0.73
 lavfi.ssim.Y=0.80
-lavfi.ssim.U=0.72
+frame:4    pts:4       pts_time:4
 lavfi.ssim.V=0.68
-lavfi.ssim.All=0.75
-lavfi.ssim.dB=5.97
+lavfi.ssim.U=0.76