Message ID | 20170720202144.GG3740@nb4 |
---|---|
State | Superseded |
Headers | show |
On 20.07.2017 22:21, Michael Niedermayer wrote: > On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote: > [...] >> fate/filter-video.mak | 16 ++++++++++++++ >> ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++++++++++++++++ >> ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++++++++++++++++ >> ref/fate/filter-refcmp-ssim-rgb | 30 ++++++++++++++++++++++++++ >> ref/fate/filter-refcmp-ssim-yuv | 30 ++++++++++++++++++++++++++ >> 5 files changed, 166 insertions(+) >> ba25d086250183b279c65419343103872074dab8 0001-fate-add-tests-for-psnr-and-ssim-filter.patch >> From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001 >> From: Tobias Rapp <t.rapp@noa-archive.com> >> Date: Tue, 11 Jul 2017 12:38:06 +0200 >> Subject: [PATCH] fate: add tests for psnr and ssim filter >> >> Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com> > > on x86-32: > > --- tests/ref/fate/filter-refcmp-psnr-yuv 2017-07-20 22:05:14.887394601 +0200 > +++ tests/data/fate/filter-refcmp-psnr-yuv 2017-07-20 22:14:51.363406746 +0200 > @@ -3,43 +3,43 @@ > lavfi.psnr.psnr.y=24.67 > lavfi.psnr.mse.u=339.38 > lavfi.psnr.psnr.u=22.82 > -lavfi.psnr.mse.v=705.41 > +lavfi.psnr.mse.v=705.31 > lavfi.psnr.psnr.v=19.65 > -lavfi.psnr.mse_avg=372.23 > +lavfi.psnr.mse_avg=372.20 > lavfi.psnr.psnr_avg=22.42 > frame:1 pts:1 pts_time:1 > -lavfi.psnr.mse.y=236.74 > +lavfi.psnr.mse.y=236.76 > lavfi.psnr.psnr.y=24.39 > lavfi.psnr.mse.u=416.17 > lavfi.psnr.psnr.u=21.94 > -lavfi.psnr.mse.v=704.98 > +lavfi.psnr.mse.v=704.97 > lavfi.psnr.psnr.v=19.65 > lavfi.psnr.mse_avg=398.66 > lavfi.psnr.psnr_avg=22.12 > frame:2 pts:2 pts_time:2 > -lavfi.psnr.mse.y=234.79 > +lavfi.psnr.mse.y=234.83 > lavfi.psnr.psnr.y=24.42 > lavfi.psnr.mse.u=435.72 > lavfi.psnr.psnr.u=21.74 > lavfi.psnr.mse.v=699.60 > lavfi.psnr.psnr.v=19.68 > -lavfi.psnr.mse_avg=401.23 > +lavfi.psnr.mse_avg=401.24 > lavfi.psnr.psnr_avg=22.10 > frame:3 pts:3 pts_time:3 > -lavfi.psnr.mse.y=250.88 > +lavfi.psnr.mse.y=250.91 > lavfi.psnr.psnr.y=24.14 > lavfi.psnr.mse.u=479.73 > lavfi.psnr.psnr.u=21.32 > -lavfi.psnr.mse.v=707.55 > +lavfi.psnr.mse.v=707.54 > lavfi.psnr.psnr.v=19.63 > -lavfi.psnr.mse_avg=422.26 > -lavfi.psnr.psnr_avg=21.88 > +lavfi.psnr.mse_avg=422.27 > +lavfi.psnr.psnr_avg=21.87 > frame:4 pts:4 pts_time:4 > -lavfi.psnr.mse.y=241.05 > +lavfi.psnr.mse.y=241.04 > lavfi.psnr.psnr.y=24.31 > lavfi.psnr.mse.u=505.04 > lavfi.psnr.psnr.u=21.10 > lavfi.psnr.mse.v=716.00 > lavfi.psnr.psnr.v=19.58 > -lavfi.psnr.mse_avg=425.79 > +lavfi.psnr.mse_avg=425.78 > lavfi.psnr.psnr_avg=21.84 > Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details. > make: *** [fate-filter-refcmp-psnr-yuv] Error 1 > make: *** Waiting for unfinished jobs.... > --- tests/ref/fate/filter-refcmp-psnr-rgb 2017-07-20 22:05:14.887394601 +0200 > +++ tests/data/fate/filter-refcmp-psnr-rgb 2017-07-20 22:14:51.367406746 +0200 > @@ -1,11 +1,11 @@ > frame:0 pts:0 pts_time:0 > -lavfi.psnr.mse.r=1381.80 > +lavfi.psnr.mse.r=1381.38 > lavfi.psnr.psnr.r=16.73 > lavfi.psnr.mse.g=896.00 > lavfi.psnr.psnr.g=18.61 > lavfi.psnr.mse.b=277.38 > lavfi.psnr.psnr.b=23.70 > -lavfi.psnr.mse_avg=851.73 > +lavfi.psnr.mse_avg=851.59 > lavfi.psnr.psnr_avg=18.83 > frame:1 pts:1 pts_time:1 > lavfi.psnr.mse.r=1380.37 > @@ -17,9 +17,9 @@ > lavfi.psnr.mse_avg=930.67 > lavfi.psnr.psnr_avg=18.44 > frame:2 pts:2 pts_time:2 > -lavfi.psnr.mse.r=1403.20 > +lavfi.psnr.mse.r=1403.21 > lavfi.psnr.psnr.r=16.66 > -lavfi.psnr.mse.g=954.05 > +lavfi.psnr.mse.g=954.04 > lavfi.psnr.psnr.g=18.34 > lavfi.psnr.mse.b=494.22 > lavfi.psnr.psnr.b=21.19 > Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details. > make: *** [fate-filter-refcmp-psnr-rgb] Error 1 > So I assume the lesson to be learned for me should be that it is not safe to compare string-ified floats and a sane test should do something like "abs(ref_val - curr_val) < eps" for each metadata entry? Regards, Tobias
On Fri, Jul 21, 2017 at 04:39:37PM +0200, Tobias Rapp wrote: > On 20.07.2017 22:21, Michael Niedermayer wrote: > >On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote: > >[...] > >> fate/filter-video.mak | 16 ++++++++++++++ > >> ref/fate/filter-refcmp-psnr-rgb | 45 ++++++++++++++++++++++++++++++++++++++++ > >> ref/fate/filter-refcmp-psnr-yuv | 45 ++++++++++++++++++++++++++++++++++++++++ > >> ref/fate/filter-refcmp-ssim-rgb | 30 ++++++++++++++++++++++++++ > >> ref/fate/filter-refcmp-ssim-yuv | 30 ++++++++++++++++++++++++++ > >> 5 files changed, 166 insertions(+) > >>ba25d086250183b279c65419343103872074dab8 0001-fate-add-tests-for-psnr-and-ssim-filter.patch > >>From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001 > >>From: Tobias Rapp <t.rapp@noa-archive.com> > >>Date: Tue, 11 Jul 2017 12:38:06 +0200 > >>Subject: [PATCH] fate: add tests for psnr and ssim filter > >> > >>Signed-off-by: Tobias Rapp <t.rapp@noa-archive.com> > > > >on x86-32: > > > >--- tests/ref/fate/filter-refcmp-psnr-yuv 2017-07-20 22:05:14.887394601 +0200 > >+++ tests/data/fate/filter-refcmp-psnr-yuv 2017-07-20 22:14:51.363406746 +0200 > >@@ -3,43 +3,43 @@ > > lavfi.psnr.psnr.y=24.67 > > lavfi.psnr.mse.u=339.38 > > lavfi.psnr.psnr.u=22.82 > >-lavfi.psnr.mse.v=705.41 > >+lavfi.psnr.mse.v=705.31 > > lavfi.psnr.psnr.v=19.65 > >-lavfi.psnr.mse_avg=372.23 > >+lavfi.psnr.mse_avg=372.20 > > lavfi.psnr.psnr_avg=22.42 > > frame:1 pts:1 pts_time:1 > >-lavfi.psnr.mse.y=236.74 > >+lavfi.psnr.mse.y=236.76 > > lavfi.psnr.psnr.y=24.39 > > lavfi.psnr.mse.u=416.17 > > lavfi.psnr.psnr.u=21.94 > >-lavfi.psnr.mse.v=704.98 > >+lavfi.psnr.mse.v=704.97 > > lavfi.psnr.psnr.v=19.65 > > lavfi.psnr.mse_avg=398.66 > > lavfi.psnr.psnr_avg=22.12 > > frame:2 pts:2 pts_time:2 > >-lavfi.psnr.mse.y=234.79 > >+lavfi.psnr.mse.y=234.83 > > lavfi.psnr.psnr.y=24.42 > > lavfi.psnr.mse.u=435.72 > > lavfi.psnr.psnr.u=21.74 > > lavfi.psnr.mse.v=699.60 > > lavfi.psnr.psnr.v=19.68 > >-lavfi.psnr.mse_avg=401.23 > >+lavfi.psnr.mse_avg=401.24 > > lavfi.psnr.psnr_avg=22.10 > > frame:3 pts:3 pts_time:3 > >-lavfi.psnr.mse.y=250.88 > >+lavfi.psnr.mse.y=250.91 > > lavfi.psnr.psnr.y=24.14 > > lavfi.psnr.mse.u=479.73 > > lavfi.psnr.psnr.u=21.32 > >-lavfi.psnr.mse.v=707.55 > >+lavfi.psnr.mse.v=707.54 > > lavfi.psnr.psnr.v=19.63 > >-lavfi.psnr.mse_avg=422.26 > >-lavfi.psnr.psnr_avg=21.88 > >+lavfi.psnr.mse_avg=422.27 > >+lavfi.psnr.psnr_avg=21.87 > > frame:4 pts:4 pts_time:4 > >-lavfi.psnr.mse.y=241.05 > >+lavfi.psnr.mse.y=241.04 > > lavfi.psnr.psnr.y=24.31 > > lavfi.psnr.mse.u=505.04 > > lavfi.psnr.psnr.u=21.10 > > lavfi.psnr.mse.v=716.00 > > lavfi.psnr.psnr.v=19.58 > >-lavfi.psnr.mse_avg=425.79 > >+lavfi.psnr.mse_avg=425.78 > > lavfi.psnr.psnr_avg=21.84 > >Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details. > >make: *** [fate-filter-refcmp-psnr-yuv] Error 1 > >make: *** Waiting for unfinished jobs.... > >--- tests/ref/fate/filter-refcmp-psnr-rgb 2017-07-20 22:05:14.887394601 +0200 > >+++ tests/data/fate/filter-refcmp-psnr-rgb 2017-07-20 22:14:51.367406746 +0200 > >@@ -1,11 +1,11 @@ > > frame:0 pts:0 pts_time:0 > >-lavfi.psnr.mse.r=1381.80 > >+lavfi.psnr.mse.r=1381.38 > > lavfi.psnr.psnr.r=16.73 > > lavfi.psnr.mse.g=896.00 > > lavfi.psnr.psnr.g=18.61 > > lavfi.psnr.mse.b=277.38 > > lavfi.psnr.psnr.b=23.70 > >-lavfi.psnr.mse_avg=851.73 > >+lavfi.psnr.mse_avg=851.59 > > lavfi.psnr.psnr_avg=18.83 > > frame:1 pts:1 pts_time:1 > > lavfi.psnr.mse.r=1380.37 > >@@ -17,9 +17,9 @@ > > lavfi.psnr.mse_avg=930.67 > > lavfi.psnr.psnr_avg=18.44 > > frame:2 pts:2 pts_time:2 > >-lavfi.psnr.mse.r=1403.20 > >+lavfi.psnr.mse.r=1403.21 > > lavfi.psnr.psnr.r=16.66 > >-lavfi.psnr.mse.g=954.05 > >+lavfi.psnr.mse.g=954.04 > > lavfi.psnr.psnr.g=18.34 > > lavfi.psnr.mse.b=494.22 > > lavfi.psnr.psnr.b=21.19 > >Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details. > >make: *** [fate-filter-refcmp-psnr-rgb] Error 1 > > > > So I assume the lesson to be learned for me should be that it is not > safe to compare string-ified floats and a sane test should do > something like "abs(ref_val - curr_val) < eps" for each metadata > entry? we might be able to get away with strings and lower precission. but yes otherwise with inexact code like floats checking the difference would likely be needed [...]
Le quartidi 4 thermidor, an CCXXV, Michael Niedermayer a écrit :
> we might be able to get away with strings and lower precission.
We might be lucky like that, but we cannot expect it to work always: the
value always has a small chance to be too near the cutoff value between
two rounded numbers.
Also, decreasing the precision increases the risk of missing a
regression.
I just wrote the attached script, it may or may not be of use.
Regards,
--- tests/ref/fate/filter-refcmp-psnr-yuv 2017-07-20 22:05:14.887394601 +0200 +++ tests/data/fate/filter-refcmp-psnr-yuv 2017-07-20 22:14:51.363406746 +0200 @@ -3,43 +3,43 @@ lavfi.psnr.psnr.y=24.67 lavfi.psnr.mse.u=339.38 lavfi.psnr.psnr.u=22.82 -lavfi.psnr.mse.v=705.41 +lavfi.psnr.mse.v=705.31 lavfi.psnr.psnr.v=19.65 -lavfi.psnr.mse_avg=372.23 +lavfi.psnr.mse_avg=372.20 lavfi.psnr.psnr_avg=22.42 frame:1 pts:1 pts_time:1 -lavfi.psnr.mse.y=236.74 +lavfi.psnr.mse.y=236.76 lavfi.psnr.psnr.y=24.39 lavfi.psnr.mse.u=416.17 lavfi.psnr.psnr.u=21.94 -lavfi.psnr.mse.v=704.98 +lavfi.psnr.mse.v=704.97 lavfi.psnr.psnr.v=19.65 lavfi.psnr.mse_avg=398.66 lavfi.psnr.psnr_avg=22.12 frame:2 pts:2 pts_time:2 -lavfi.psnr.mse.y=234.79 +lavfi.psnr.mse.y=234.83 lavfi.psnr.psnr.y=24.42 lavfi.psnr.mse.u=435.72 lavfi.psnr.psnr.u=21.74 lavfi.psnr.mse.v=699.60 lavfi.psnr.psnr.v=19.68 -lavfi.psnr.mse_avg=401.23 +lavfi.psnr.mse_avg=401.24 lavfi.psnr.psnr_avg=22.10 frame:3 pts:3 pts_time:3 -lavfi.psnr.mse.y=250.88 +lavfi.psnr.mse.y=250.91 lavfi.psnr.psnr.y=24.14 lavfi.psnr.mse.u=479.73 lavfi.psnr.psnr.u=21.32 -lavfi.psnr.mse.v=707.55 +lavfi.psnr.mse.v=707.54 lavfi.psnr.psnr.v=19.63 -lavfi.psnr.mse_avg=422.26 -lavfi.psnr.psnr_avg=21.88 +lavfi.psnr.mse_avg=422.27 +lavfi.psnr.psnr_avg=21.87 frame:4 pts:4 pts_time:4 -lavfi.psnr.mse.y=241.05 +lavfi.psnr.mse.y=241.04 lavfi.psnr.psnr.y=24.31 lavfi.psnr.mse.u=505.04 lavfi.psnr.psnr.u=21.10 lavfi.psnr.mse.v=716.00 lavfi.psnr.psnr.v=19.58 -lavfi.psnr.mse_avg=425.79 +lavfi.psnr.mse_avg=425.78 lavfi.psnr.psnr_avg=21.84 Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details. make: *** [fate-filter-refcmp-psnr-yuv] Error 1 make: *** Waiting for unfinished jobs.... --- tests/ref/fate/filter-refcmp-psnr-rgb 2017-07-20 22:05:14.887394601 +0200 +++ tests/data/fate/filter-refcmp-psnr-rgb 2017-07-20 22:14:51.367406746 +0200 @@ -1,11 +1,11 @@ frame:0 pts:0 pts_time:0 -lavfi.psnr.mse.r=1381.80 +lavfi.psnr.mse.r=1381.38 lavfi.psnr.psnr.r=16.73 lavfi.psnr.mse.g=896.00 lavfi.psnr.psnr.g=18.61 lavfi.psnr.mse.b=277.38 lavfi.psnr.psnr.b=23.70 -lavfi.psnr.mse_avg=851.73 +lavfi.psnr.mse_avg=851.59 lavfi.psnr.psnr_avg=18.83 frame:1 pts:1 pts_time:1 lavfi.psnr.mse.r=1380.37 @@ -17,9 +17,9 @@ lavfi.psnr.mse_avg=930.67 lavfi.psnr.psnr_avg=18.44 frame:2 pts:2 pts_time:2 -lavfi.psnr.mse.r=1403.20 +lavfi.psnr.mse.r=1403.21 lavfi.psnr.psnr.r=16.66 -lavfi.psnr.mse.g=954.05 +lavfi.psnr.mse.g=954.04 lavfi.psnr.psnr.g=18.34 lavfi.psnr.mse.b=494.22 lavfi.psnr.psnr.b=21.19