[FFmpeg-devel,3/3] fate: add loudnorm filter test

Submitted by Marton Balint on Oct. 16, 2016, 7:32 p.m.

Details

Message ID 1476646324-15429-3-git-send-email-cus@passwd.hu
State New
Headers show

Commit Message

Marton Balint Oct. 16, 2016, 7:32 p.m.
Signed-off-by: Marton Balint <cus@passwd.hu>
---
 tests/fate/filter-audio.mak | 7 +++++++
 1 file changed, 7 insertions(+)

Comments

Marton Balint Oct. 16, 2016, 8:12 p.m.
On Sun, 16 Oct 2016, Marton Balint wrote:

> Signed-off-by: Marton Balint <cus@passwd.hu>
> ---
> tests/fate/filter-audio.mak | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
> index 9c6f7cd..d376f25 100644
> --- a/tests/fate/filter-audio.mak
> +++ b/tests/fate/filter-audio.mak
> @@ -279,6 +279,13 @@ fate-filter-hdcd-detect-errors: CMD = md5 -i $(SRC) -af hdcd -f s24le
> fate-filter-hdcd-detect-errors: CMP = grep
> fate-filter-hdcd-detect-errors: REF = detectable errors: [1-9]
>
> +FATE_AFILTER-$(call FILTERDEMDECENCMUX, LOUDNORM, AAC, AAC, PCM_S16LE, PCM_S16LE) += fate-filter-loudnorm-simple
> +fate-filter-loudnorm-simple: SRC = $(SAMPLES)/aac/sintel.aac
> +fate-filter-loudnorm-simple: CMD = ffmpeg -t 30 -i $(SRC) -af loudnorm=i=-23 -f s16le -ar 44100 -
> +fate-filter-loudnorm-simple: REF = $(SAMPLES)/filter/loudnorm-simple.pcm
> +fate-filter-loudnorm-simple: CMP = oneoff
> +fate-filter-loudnorm-simple: CMP_UNIT = s16
> +

This patch needs two files in the fate samples:

The audio part of the Sintel movie, as a source, because I wanted to test 
with a real world example, with proper length. And the reference 
file. Sources can be generated like this:

wget http://media.xiph.org/sintel/sintel-master-st.flac
ffmpeg -i sintel-master-st.flac -codec aac -b 96k fate-suite/aac/sintel.aac
ffmpeg -t 30 -i fate-suite/aac/sintel.aac -af loudnorm=i=-23 -f s16le -ar 44100 fate-suite/filter/loudnorm-simple.pcm

Due to the 96k AAC codec, sintel.aac is about 15M, loudnorm-simple.pcm is 
around 6M. Still a bit big, but IMHO it is ok to add these to the 
fate samples.

Regards,
Marton
Michael Niedermayer Oct. 17, 2016, 11:39 p.m.
On Sun, Oct 16, 2016 at 10:12:17PM +0200, Marton Balint wrote:
> 
> On Sun, 16 Oct 2016, Marton Balint wrote:
> 
> >Signed-off-by: Marton Balint <cus@passwd.hu>
> >---
> >tests/fate/filter-audio.mak | 7 +++++++
> >1 file changed, 7 insertions(+)
> >
> >diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
> >index 9c6f7cd..d376f25 100644
> >--- a/tests/fate/filter-audio.mak
> >+++ b/tests/fate/filter-audio.mak
> >@@ -279,6 +279,13 @@ fate-filter-hdcd-detect-errors: CMD = md5 -i $(SRC) -af hdcd -f s24le
> >fate-filter-hdcd-detect-errors: CMP = grep
> >fate-filter-hdcd-detect-errors: REF = detectable errors: [1-9]
> >
> >+FATE_AFILTER-$(call FILTERDEMDECENCMUX, LOUDNORM, AAC, AAC, PCM_S16LE, PCM_S16LE) += fate-filter-loudnorm-simple
> >+fate-filter-loudnorm-simple: SRC = $(SAMPLES)/aac/sintel.aac
> >+fate-filter-loudnorm-simple: CMD = ffmpeg -t 30 -i $(SRC) -af loudnorm=i=-23 -f s16le -ar 44100 -
> >+fate-filter-loudnorm-simple: REF = $(SAMPLES)/filter/loudnorm-simple.pcm
> >+fate-filter-loudnorm-simple: CMP = oneoff
> >+fate-filter-loudnorm-simple: CMP_UNIT = s16
> >+
> 
> This patch needs two files in the fate samples:
> 
> The audio part of the Sintel movie, as a source, because I wanted to
> test with a real world example, with proper length. And the
> reference file. Sources can be generated like this:
> 
> wget http://media.xiph.org/sintel/sintel-master-st.flac
> ffmpeg -i sintel-master-st.flac -codec aac -b 96k fate-suite/aac/sintel.aac
> ffmpeg -t 30 -i fate-suite/aac/sintel.aac -af loudnorm=i=-23 -f s16le -ar 44100 fate-suite/filter/loudnorm-simple.pcm
> 
> Due to the 96k AAC codec, sintel.aac is about 15M,

are low bitrate speech codecs unsuitable instead of aac for this ?
that would cut the size down by alot


[...]
Marton Balint Oct. 18, 2016, 12:20 a.m.
On Tue, 18 Oct 2016, Michael Niedermayer wrote:

> On Sun, Oct 16, 2016 at 10:12:17PM +0200, Marton Balint wrote:
>>
>> On Sun, 16 Oct 2016, Marton Balint wrote:
>>
>>> Signed-off-by: Marton Balint <cus@passwd.hu>
>>> ---
>>> tests/fate/filter-audio.mak | 7 +++++++
>>> 1 file changed, 7 insertions(+)
>>>
>>> diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
>>> index 9c6f7cd..d376f25 100644
>>> --- a/tests/fate/filter-audio.mak
>>> +++ b/tests/fate/filter-audio.mak
>>> @@ -279,6 +279,13 @@ fate-filter-hdcd-detect-errors: CMD = md5 -i $(SRC) -af hdcd -f s24le
>>> fate-filter-hdcd-detect-errors: CMP = grep
>>> fate-filter-hdcd-detect-errors: REF = detectable errors: [1-9]
>>>
>>> +FATE_AFILTER-$(call FILTERDEMDECENCMUX, LOUDNORM, AAC, AAC, PCM_S16LE, PCM_S16LE) += fate-filter-loudnorm-simple
>>> +fate-filter-loudnorm-simple: SRC = $(SAMPLES)/aac/sintel.aac
>>> +fate-filter-loudnorm-simple: CMD = ffmpeg -t 30 -i $(SRC) -af loudnorm=i=-23 -f s16le -ar 44100 -
>>> +fate-filter-loudnorm-simple: REF = $(SAMPLES)/filter/loudnorm-simple.pcm
>>> +fate-filter-loudnorm-simple: CMP = oneoff
>>> +fate-filter-loudnorm-simple: CMP_UNIT = s16
>>> +
>>
>> This patch needs two files in the fate samples:
>>
>> The audio part of the Sintel movie, as a source, because I wanted to
>> test with a real world example, with proper length. And the
>> reference file. Sources can be generated like this:
>>
>> wget http://media.xiph.org/sintel/sintel-master-st.flac
>> ffmpeg -i sintel-master-st.flac -codec aac -b 96k fate-suite/aac/sintel.aac
>> ffmpeg -t 30 -i fate-suite/aac/sintel.aac -af loudnorm=i=-23 -f s16le -ar 44100 fate-suite/filter/loudnorm-simple.pcm
>>
>> Due to the 96k AAC codec, sintel.aac is about 15M,
>
> are low bitrate speech codecs unsuitable instead of aac for this ?
> that would cut the size down by alot

In theory, maybe, on the other hand, we are only using the first 30 second 
of the sample, so if size is an issue, we can reduce it to around 500k and 
the fate test will still work.

Since the reference file alone is 6M, it does not seem to make too much 
difference if the sample is 500k, or less, so I'd prefer the 
normal codec. I am not sure I can give you a pure technical reasoning, the 
only thing I could think of is that as far as I know a speech codec is 
usually not good at very low or very high frequencies, but it is a 
good idea to test loudness measurement with all kind of frequencies, 
because of it's frequency dependant filters.

Regards,
Marton
Michael Niedermayer Oct. 18, 2016, 2 a.m.
On Tue, Oct 18, 2016 at 02:20:03AM +0200, Marton Balint wrote:
> 
> On Tue, 18 Oct 2016, Michael Niedermayer wrote:
> 
> >On Sun, Oct 16, 2016 at 10:12:17PM +0200, Marton Balint wrote:
> >>
> >>On Sun, 16 Oct 2016, Marton Balint wrote:
> >>
> >>>Signed-off-by: Marton Balint <cus@passwd.hu>
> >>>---
> >>>tests/fate/filter-audio.mak | 7 +++++++
> >>>1 file changed, 7 insertions(+)
> >>>
> >>>diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
> >>>index 9c6f7cd..d376f25 100644
> >>>--- a/tests/fate/filter-audio.mak
> >>>+++ b/tests/fate/filter-audio.mak
> >>>@@ -279,6 +279,13 @@ fate-filter-hdcd-detect-errors: CMD = md5 -i $(SRC) -af hdcd -f s24le
> >>>fate-filter-hdcd-detect-errors: CMP = grep
> >>>fate-filter-hdcd-detect-errors: REF = detectable errors: [1-9]
> >>>
> >>>+FATE_AFILTER-$(call FILTERDEMDECENCMUX, LOUDNORM, AAC, AAC, PCM_S16LE, PCM_S16LE) += fate-filter-loudnorm-simple
> >>>+fate-filter-loudnorm-simple: SRC = $(SAMPLES)/aac/sintel.aac
> >>>+fate-filter-loudnorm-simple: CMD = ffmpeg -t 30 -i $(SRC) -af loudnorm=i=-23 -f s16le -ar 44100 -
> >>>+fate-filter-loudnorm-simple: REF = $(SAMPLES)/filter/loudnorm-simple.pcm
> >>>+fate-filter-loudnorm-simple: CMP = oneoff
> >>>+fate-filter-loudnorm-simple: CMP_UNIT = s16
> >>>+
> >>
> >>This patch needs two files in the fate samples:
> >>
> >>The audio part of the Sintel movie, as a source, because I wanted to
> >>test with a real world example, with proper length. And the
> >>reference file. Sources can be generated like this:
> >>
> >>wget http://media.xiph.org/sintel/sintel-master-st.flac
> >>ffmpeg -i sintel-master-st.flac -codec aac -b 96k fate-suite/aac/sintel.aac
> >>ffmpeg -t 30 -i fate-suite/aac/sintel.aac -af loudnorm=i=-23 -f s16le -ar 44100 fate-suite/filter/loudnorm-simple.pcm
> >>
> >>Due to the 96k AAC codec, sintel.aac is about 15M,
> >
> >are low bitrate speech codecs unsuitable instead of aac for this ?
> >that would cut the size down by alot
> 
> In theory, maybe, on the other hand, we are only using the first 30
> second of the sample, so if size is an issue, we can reduce it to
> around 500k and the fate test will still work.
> 
> Since the reference file alone is 6M, it does not seem to make too
> much difference if the sample is 500k, or less, so I'd prefer the
> normal codec. I am not sure I can give you a pure technical
> reasoning, the only thing I could think of is that as far as I know
> a speech codec is usually not good at very low or very high

> frequencies, but it is a good idea to test loudness measurement with
> all kind of frequencies, because of it's frequency dependant
> filters.

a synthetic sample that contains a sine sweep would excercise all
frequencies
tests/audiogen would need very little space to generate such sample
but i dont know if its worse for testing ?

also is using a PCM file optimal as reference ?
it may be hard to understand from the psnr difference what exactly
changed if some code change triggers a change in the output.
and also if the code is changed to require a new reference in the
future we would need to keep multiple reference files

either way, if you say that these files are the best solution then ill
upload them (assuming the new fate tests pass, didnt try yet)

[...]

Patch hide | download patch | download mbox

diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
index 9c6f7cd..d376f25 100644
--- a/tests/fate/filter-audio.mak
+++ b/tests/fate/filter-audio.mak
@@ -279,6 +279,13 @@  fate-filter-hdcd-detect-errors: CMD = md5 -i $(SRC) -af hdcd -f s24le
 fate-filter-hdcd-detect-errors: CMP = grep
 fate-filter-hdcd-detect-errors: REF = detectable errors: [1-9]
 
+FATE_AFILTER-$(call FILTERDEMDECENCMUX, LOUDNORM, AAC, AAC, PCM_S16LE, PCM_S16LE) += fate-filter-loudnorm-simple
+fate-filter-loudnorm-simple: SRC = $(SAMPLES)/aac/sintel.aac
+fate-filter-loudnorm-simple: CMD = ffmpeg -t 30 -i $(SRC) -af loudnorm=i=-23 -f s16le -ar 44100 -
+fate-filter-loudnorm-simple: REF = $(SAMPLES)/filter/loudnorm-simple.pcm
+fate-filter-loudnorm-simple: CMP = oneoff
+fate-filter-loudnorm-simple: CMP_UNIT = s16
+
 FATE_AFILTER-yes += fate-filter-formats
 fate-filter-formats: libavfilter/tests/formats$(EXESUF)
 fate-filter-formats: CMD = run libavfilter/tests/formats