diff mbox

[FFmpeg-devel,PATCHv3] af_hdcd: more FATE tests

Message ID 20161218184845.26928-1-pburt0@gmail.com
State Superseded
Headers show

Commit Message

Burt P Dec. 18, 2016, 6:48 p.m. UTC 
Additional/Modified FATE tests improve code coverage from 63.7% to 98.1%.

Changed fate-suite sample files:
* filter/hdcd-encoding-errors.flac (1.3M) replaced by
  a smaller version (140K). It can be replaced because the test
  only looks for a non-zero number of errors, so the existing test
  will still pass.
* filter/hdcd-mix.flac (958K) added. It is a much better test than
  filter/hdcd.flac (910K), which is now unused, but can't be removed.
* filter/hdcd-fake20bit.flac (168K) added. It is the first second of
  filter/hdcd.flac, with the 16-bit LSB copied into bit 20 of a 24-bit
  stream. There isn't an actual non-16-bit HDCD sample available to test.
Net change -34K. Would be -944K if hdcd.flac could be removed.

Signed-off-by: Burt P <pburt0@gmail.com>
---
 tests/fate/filter-audio.mak | 86 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 74 insertions(+), 12 deletions(-)

Comments

Michael Niedermayer Dec. 19, 2016, 12:54 a.m. UTC | #1 
On Sun, Dec 18, 2016 at 12:48:45PM -0600, Burt P wrote:
> Additional/Modified FATE tests improve code coverage from 63.7% to 98.1%.
> 
> Changed fate-suite sample files:
> * filter/hdcd-encoding-errors.flac (1.3M) replaced by
>   a smaller version (140K). It can be replaced because the test
>   only looks for a non-zero number of errors, so the existing test
>   will still pass.

IMO replacing files is not ok
it would change all past instances of the related fate test
a bug report refering to a fate failure could become unreproduceable
or otherwise working fate tests could start failing ...
as much as i prefer to safe a few bytes in this case i prefer to waste
some space over the potential problems

you can add files, but not remove or replace unless they are truly
unused by every past checkout

[...]
Burt P Dec. 20, 2016, 4:27 a.m. UTC | #2 
On Sun, Dec 18, 2016 at 6:54 PM, Michael Niedermayer
<michael@niedermayer.cc> wrote:
> On Sun, Dec 18, 2016 at 12:48:45PM -0600, Burt P wrote:
>> Additional/Modified FATE tests improve code coverage from 63.7% to 98.1%.
>>
>> Changed fate-suite sample files:
>> * filter/hdcd-encoding-errors.flac (1.3M) replaced by
>>   a smaller version (140K). It can be replaced because the test
>>   only looks for a non-zero number of errors, so the existing test
>>   will still pass.
>
> IMO replacing files is not ok
> it would change all past instances of the related fate test
> a bug report refering to a fate failure could become unreproduceable
> or otherwise working fate tests could start failing ...
> as much as i prefer to safe a few bytes in this case i prefer to waste
> some space over the potential problems
>
> you can add files, but not remove or replace unless they are truly
> unused by every past checkout

Well, I don't mind if it is kept, I was just trying to offset the additions.
I understand why removing or replacing samples is generally not a good
idea, but I would point out that in this case, the file could be replaced
without changing the result in any past revision of the test.
This particular test outcome is determined by grepping a line out of the
log that reports the number of errors encountered, and it only has to be
non-zero, so the shorter sample with fewer errors (but at least one) still
passes any old revision.
Michael Niedermayer Dec. 20, 2016, 5:57 p.m. UTC | #3 
On Mon, Dec 19, 2016 at 10:27:20PM -0600, Burt P. wrote:
> On Sun, Dec 18, 2016 at 6:54 PM, Michael Niedermayer
> <michael@niedermayer.cc> wrote:
> > On Sun, Dec 18, 2016 at 12:48:45PM -0600, Burt P wrote:
> >> Additional/Modified FATE tests improve code coverage from 63.7% to 98.1%.
> >>
> >> Changed fate-suite sample files:
> >> * filter/hdcd-encoding-errors.flac (1.3M) replaced by
> >>   a smaller version (140K). It can be replaced because the test
> >>   only looks for a non-zero number of errors, so the existing test
> >>   will still pass.
> >
> > IMO replacing files is not ok
> > it would change all past instances of the related fate test
> > a bug report refering to a fate failure could become unreproduceable
> > or otherwise working fate tests could start failing ...
> > as much as i prefer to safe a few bytes in this case i prefer to waste
> > some space over the potential problems
> >
> > you can add files, but not remove or replace unless they are truly
> > unused by every past checkout
> 
> Well, I don't mind if it is kept, I was just trying to offset the additions.
> I understand why removing or replacing samples is generally not a good
> idea, but I would point out that in this case, the file could be replaced
> without changing the result in any past revision of the test.
> This particular test outcome is determined by grepping a line out of the
> log that reports the number of errors encountered, and it only has to be
> non-zero, so the shorter sample with fewer errors (but at least one) still
> passes any old revision.

did you post links to any of the files or how can the new files be
added ?

[...]
Burt P Dec. 26, 2016, 6:27 p.m. UTC | #4 
On Tue, Dec 20, 2016 at 11:57 AM, Michael Niedermayer
<michael@niedermayer.cc> wrote:
> did you post links to any of the files or how can the new files be
> added ?

Yes, in IRC. Thank you for uploading.
diff mbox

Patch

diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
index 9c6f7cd..b356bcc 100644
--- a/tests/fate/filter-audio.mak
+++ b/tests/fate/filter-audio.mak
@@ -255,17 +255,55 @@  fate-filter-volume: CMD = md5 -i $(SRC) -af aperms=random,volume=precision=fixed
 fate-filter-volume: CMP = oneline
 fate-filter-volume: REF = 4d6ba75ef3e32d305d066b9bc771d6f4
 
-FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd
-fate-filter-hdcd: SRC = $(TARGET_SAMPLES)/filter/hdcd.flac
-fate-filter-hdcd: CMD = md5 -i $(SRC) -af hdcd -f s24le
-fate-filter-hdcd: CMP = oneline
-fate-filter-hdcd: REF = 5db465a58d2fd0d06ca944b883b33476
-
-FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze
-fate-filter-hdcd-analyze: SRC = $(TARGET_SAMPLES)/filter/hdcd.flac
-fate-filter-hdcd-analyze: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=pe -f s24le
-fate-filter-hdcd-analyze: CMP = oneline
-fate-filter-hdcd-analyze: REF = 6e39dc4629c1e84321c0d8bc069b42f6
+# hdcd-mix.flac is a mix of three different sources which are interesting for various reasons:
+# first 5 seconds uses packet format A and max LLE of -7.0db
+# second 5 seconds uses packet format B and has a gain mismatch between channels
+# last 10 seconds is not HDCD but has a coincidental HDCD packet, it needs to be 10 seconds because it also tests the cdt expiration
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-mix
+fate-filter-hdcd-mix: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-mix: CMD = md5 -i $(SRC) -af hdcd -f s24le
+fate-filter-hdcd-mix: CMP = oneline
+fate-filter-hdcd-mix: REF = e7079913e90c124460cdbc712df5b84c
+
+# output will be different because of the gain mismatch in the second and third parts
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-mix-psoff
+fate-filter-hdcd-mix-psoff: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-mix-psoff: CMD = md5 -i $(SRC) -af hdcd=process_stereo=false -f s24le
+fate-filter-hdcd-mix-psoff: CMP = oneline
+fate-filter-hdcd-mix-psoff: REF = bd0e81fe17696b825ee3515ab928e6bb
+
+# test the different analyze modes
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze-pe
+fate-filter-hdcd-analyze-pe: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-analyze-pe: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=pe -f s24le
+fate-filter-hdcd-analyze-pe: CMP = oneline
+fate-filter-hdcd-analyze-pe: REF = bb83e97bbd0064b9b1c0ef2f2c8f0c77
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze-lle
+fate-filter-hdcd-analyze-lle: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-analyze-lle: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=lle -f s24le
+fate-filter-hdcd-analyze-lle: CMP = oneline
+fate-filter-hdcd-analyze-lle: REF = 121cc4a681aa0caef5c664fece7a3ddc
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze-cdt
+fate-filter-hdcd-analyze-cdt: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-analyze-cdt: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=cdt -f s24le
+fate-filter-hdcd-analyze-cdt: CMP = oneline
+fate-filter-hdcd-analyze-cdt: REF = 12136e6a00dd532994f6edcc347af1d4
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze-tgm
+fate-filter-hdcd-analyze-tgm: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-analyze-tgm: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=tgm -f s24le
+fate-filter-hdcd-analyze-tgm: CMP = oneline
+fate-filter-hdcd-analyze-tgm: REF = a3c39f62e9b9b42c9c440d0045d5fb2f
+# the two additional analyze modes from libhdcd
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze-ltgm
+fate-filter-hdcd-analyze-ltgm: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-analyze-ltgm: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=lle:process_stereo=false -f s24le
+fate-filter-hdcd-analyze-ltgm: CMP = oneline
+fate-filter-hdcd-analyze-ltgm: REF = 76ffd86b762b5a93332039f27e4c0c0e
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-analyze-pel
+fate-filter-hdcd-analyze-pel: SRC = $(TARGET_SAMPLES)/filter/hdcd-mix.flac
+fate-filter-hdcd-analyze-pel: CMD = md5 -i $(SRC) -af hdcd=analyze_mode=pe:force_pe=true -f s24le
+fate-filter-hdcd-analyze-pel: CMP = oneline
+fate-filter-hdcd-analyze-pel: REF = 8156c5a3658d789ab46447d62151f5e9
 
 FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-false-positive
 fate-filter-hdcd-false-positive: SRC = $(TARGET_SAMPLES)/filter/hdcd-false-positive.flac
@@ -274,11 +312,35 @@  fate-filter-hdcd-false-positive: CMP = grep
 fate-filter-hdcd-false-positive: REF = HDCD detected: no
 
 FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-detect-errors
-fate-filter-hdcd-detect-errors: SRC = $(TARGET_SAMPLES)/filter/hdcd-encoding-errors.flac
+fate-filter-hdcd-detect-errors: SRC = $(TARGET_SAMPLES)/filter/hdcd-encoding-errors2.flac
 fate-filter-hdcd-detect-errors: CMD = md5 -i $(SRC) -af hdcd -f s24le
 fate-filter-hdcd-detect-errors: CMP = grep
 fate-filter-hdcd-detect-errors: REF = detectable errors: [1-9]
 
+# 20bit HDCD
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, FLAC, FLAC, PCM_S32LE, PCM_S32LE) += fate-filter-hdcd-20bit
+fate-filter-hdcd-20bit: SRC = $(TARGET_SAMPLES)/filter/hdcd-fake20bit.flac
+fate-filter-hdcd-20bit: CMD = md5 -i $(SRC) -af hdcd=bits_per_sample=20 -f s32le
+fate-filter-hdcd-20bit: CMP = oneline
+fate-filter-hdcd-20bit: REF = 365ded883a4a92483b15b69babc81390
+
+# non-hdcd tests of different input formats for code coverage
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, WAV, PCM_S16LE, PCM_S24LE, PCM_S24LE) += fate-filter-hdcd-mono
+fate-filter-hdcd-mono: SRC = $(TARGET_SAMPLES)/audiomatch/tones_44100_mono.wav
+fate-filter-hdcd-mono: CMD = md5 -i $(SRC) -af hdcd -f s24le
+fate-filter-hdcd-mono: CMP = oneline
+fate-filter-hdcd-mono: REF = f51b114b20728e6a463a9491c643d166
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, WV, WAVPACK, PCM_S32LE, PCM_S32LE) += fate-filter-hdcd-s16p
+fate-filter-hdcd-s16p: SRC = $(TARGET_SAMPLES)/wavpack/lossless/16bit-partial.wv
+fate-filter-hdcd-s16p: CMD = md5 -i $(SRC) -af hdcd -f s32le
+fate-filter-hdcd-s16p: CMP = oneline
+fate-filter-hdcd-s16p: REF = 4e767f436b891ac59810a8b2b1d7e96b
+FATE_AFILTER_SAMPLES-$(call FILTERDEMDECENCMUX, HDCD, WV, WAVPACK, PCM_S32LE, PCM_S32LE) += fate-filter-hdcd-s32p
+fate-filter-hdcd-s32p: SRC = $(TARGET_SAMPLES)/wavpack/lossless/24bit-partial.wv
+fate-filter-hdcd-s32p: CMD = md5 -i $(SRC) -af hdcd -f s32le
+fate-filter-hdcd-s32p: CMP = oneline
+fate-filter-hdcd-s32p: REF = 0c5513e83eedaa10ab6fac9ddc173cf5
+
 FATE_AFILTER-yes += fate-filter-formats
 fate-filter-formats: libavfilter/tests/formats$(EXESUF)
 fate-filter-formats: CMD = run libavfilter/tests/formats