diff mbox series

[FFmpeg-devel] fate/filter-video.mak: do not use bit-exact check for dnn_processing

Message ID 1579271339-24322-1-git-send-email-yejun.guo@intel.com
State Superseded
Headers show
Series [FFmpeg-devel] fate/filter-video.mak: do not use bit-exact check for dnn_processing
Related show

Checks

Context Check Description
andriy/ffmpeg-patchwork pending
andriy/ffmpeg-patchwork success Applied patch
andriy/ffmpeg-patchwork success Configure finished
andriy/ffmpeg-patchwork success Make finished

Commit Message

Guo, Yejun Jan. 17, 2020, 2:28 p.m. UTC
the reason is that the tested models are in float format

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
---
 tests/fate/filter-video.mak                        |  9 +++-
 ...filter-dnn_processing-halve_first_channel_float | 55 ----------------------
 .../fate/filter-dnn_processing-halve_gray_float    | 55 ----------------------
 3 files changed, 7 insertions(+), 112 deletions(-)
 delete mode 100644 tests/ref/fate/filter-dnn_processing-halve_first_channel_float
 delete mode 100644 tests/ref/fate/filter-dnn_processing-halve_gray_float

Comments

Carl Eugen Hoyos Jan. 17, 2020, 11:06 p.m. UTC | #1
Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun <yejun.guo@intel.com>:
>
> the reason is that the tested models are in float format

The two reference files are ~1MB together: Do we want / need that?
(No opinion here.)

Carl Eugen
Guo, Yejun Jan. 18, 2020, 1:33 p.m. UTC | #2
> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> Carl Eugen Hoyos
> Sent: Saturday, January 18, 2020 7:06 AM
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> check for dnn_processing
> 
> Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun <yejun.guo@intel.com>:
> >
> > the reason is that the tested models are in float format
> 
> The two reference files are ~1MB together: Do we want / need that?
> (No opinion here.)

how about just keeping one test (for format rgb24) for vf_dnn_processing? The rgb format might be the most popular format for vf_dnn_processing.

And the test has two materials:
1)  fate-suite/dnn_processing/dnn_rgb_processing.model (can be renamed from halve_first_channel_float.model, 184 byte now)
2)  fate-suite/filter-reference/dnn_rgb_processing.raw (can be renamed from dnn_processing-halve_first_channel_float.raw, 446KB now)

When we support more models (the most effort is to add more dnn layers in native mode),
I can update both dnn_rgb_processing.model (the file size will be increased bit by bit) and dnn_rgb_processing.raw (the file size will not change much).

> 
> Carl Eugen
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Steven Liu Jan. 18, 2020, 1:51 p.m. UTC | #3
> 在 2020年1月18日,21:33,Guo, Yejun <yejun.guo@intel.com> 写道:
> 
> 
> 
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
>> Carl Eugen Hoyos
>> Sent: Saturday, January 18, 2020 7:06 AM
>> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
>> check for dnn_processing
>> 
>> Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun <yejun.guo@intel.com>:
>>> 
>>> the reason is that the tested models are in float format
>> 
>> The two reference files are ~1MB together: Do we want / need that?
>> (No opinion here.)
> 
> how about just keeping one test (for format rgb24) for vf_dnn_processing? The rgb format might be the most popular format for vf_dnn_processing.
> 
> And the test has two materials:
> 1)  fate-suite/dnn_processing/dnn_rgb_processing.model (can be renamed from halve_first_channel_float.model, 184 byte now)
> 2)  fate-suite/filter-reference/dnn_rgb_processing.raw (can be renamed from dnn_processing-halve_first_channel_float.raw, 446KB now)
> 
> When we support more models (the most effort is to add more dnn layers in native mode),
> I can update both dnn_rgb_processing.model (the file size will be increased bit by bit) and dnn_rgb_processing.raw (the file size will not change much).
What about compress them to bz2? Uncompress them when need them?
> 
>> 
>> Carl Eugen
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Thanks
Steven
Guo, Yejun Jan. 19, 2020, 1:06 a.m. UTC | #4
> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> Steven Liu
> Sent: Saturday, January 18, 2020 9:52 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Cc: Steven Liu <lq@chinaffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> check for dnn_processing
> 
> 
> 
> > 在 2020年1月18日,21:33,Guo, Yejun <yejun.guo@intel.com> 写道:
> >
> >
> >
> >> -----Original Message-----
> >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> Of
> >> Carl Eugen Hoyos
> >> Sent: Saturday, January 18, 2020 7:06 AM
> >> To: FFmpeg development discussions and patches
> <ffmpeg-devel@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use
> bit-exact
> >> check for dnn_processing
> >>
> >> Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun
> <yejun.guo@intel.com>:
> >>>
> >>> the reason is that the tested models are in float format
> >>
> >> The two reference files are ~1MB together: Do we want / need that?
> >> (No opinion here.)
> >
> > how about just keeping one test (for format rgb24) for vf_dnn_processing?
> The rgb format might be the most popular format for vf_dnn_processing.
> >
> > And the test has two materials:
> > 1)  fate-suite/dnn_processing/dnn_rgb_processing.model (can be renamed
> from halve_first_channel_float.model, 184 byte now)
> > 2)  fate-suite/filter-reference/dnn_rgb_processing.raw (can be renamed
> from dnn_processing-halve_first_channel_float.raw, 446KB now)
> >
> > When we support more models (the most effort is to add more dnn layers in
> native mode),
> > I can update both dnn_rgb_processing.model (the file size will be increased
> bit by bit) and dnn_rgb_processing.raw (the file size will not change much).
> What about compress them to bz2? Uncompress them when need them?

thanks, and just one concern, since ffmpeg is used on many platforms,
is there always an unzip program on every platform's default setup?


> >
> >>
> >> Carl Eugen
> >> _______________________________________________
> >> ffmpeg-devel mailing list
> >> ffmpeg-devel@ffmpeg.org
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>
> >> To unsubscribe, visit link above, or email
> >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> 
> Thanks
> Steven
> 
> 
> 
> 
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Carl Eugen Hoyos Jan. 19, 2020, 6:54 p.m. UTC | #5
Am Sa., 18. Jan. 2020 um 14:52 Uhr schrieb Steven Liu <lq@chinaffmpeg.org>:
>
>
>
> > 在 2020年1月18日,21:33,Guo, Yejun <yejun.guo@intel.com> 写道:
> >
> >
> >
> >> -----Original Message-----
> >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> >> Carl Eugen Hoyos
> >> Sent: Saturday, January 18, 2020 7:06 AM
> >> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> >> check for dnn_processing
> >>
> >> Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun <yejun.guo@intel.com>:
> >>>
> >>> the reason is that the tested models are in float format
> >>
> >> The two reference files are ~1MB together: Do we want / need that?
> >> (No opinion here.)
> >
> > how about just keeping one test (for format rgb24) for vf_dnn_processing? The rgb format might be the most popular format for vf_dnn_processing.
> >
> > And the test has two materials:
> > 1)  fate-suite/dnn_processing/dnn_rgb_processing.model (can be renamed from halve_first_channel_float.model, 184 byte now)
> > 2)  fate-suite/filter-reference/dnn_rgb_processing.raw (can be renamed from dnn_processing-halve_first_channel_float.raw, 446KB now)
> >
> > When we support more models (the most effort is to add more dnn layers in native mode),
> > I can update both dnn_rgb_processing.model (the file size will be increased bit by bit) and dnn_rgb_processing.raw (the file size will not change much).
> What about compress them to bz2? Uncompress them when need them?

No, we do not require bz2 for fate and should not add more requirements.

Carl Eugen
Guo, Yejun Jan. 20, 2020, 1:15 a.m. UTC | #6
> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> Carl Eugen Hoyos
> Sent: Monday, January 20, 2020 2:55 AM
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> check for dnn_processing
> 
> Am Sa., 18. Jan. 2020 um 14:52 Uhr schrieb Steven Liu <lq@chinaffmpeg.org>:
> >
> >
> >
> > > 在 2020年1月18日,21:33,Guo, Yejun <yejun.guo@intel.com> 写道:
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> Of
> > >> Carl Eugen Hoyos
> > >> Sent: Saturday, January 18, 2020 7:06 AM
> > >> To: FFmpeg development discussions and patches
> <ffmpeg-devel@ffmpeg.org>
> > >> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use
> bit-exact
> > >> check for dnn_processing
> > >>
> > >> Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun
> <yejun.guo@intel.com>:
> > >>>
> > >>> the reason is that the tested models are in float format
> > >>
> > >> The two reference files are ~1MB together: Do we want / need that?
> > >> (No opinion here.)
> > >
> > > how about just keeping one test (for format rgb24) for vf_dnn_processing?
> The rgb format might be the most popular format for vf_dnn_processing.
> > >
> > > And the test has two materials:
> > > 1)  fate-suite/dnn_processing/dnn_rgb_processing.model (can be
> renamed from halve_first_channel_float.model, 184 byte now)
> > > 2)  fate-suite/filter-reference/dnn_rgb_processing.raw (can be renamed
> from dnn_processing-halve_first_channel_float.raw, 446KB now)
> > >
> > > When we support more models (the most effort is to add more dnn layers in
> native mode),
> > > I can update both dnn_rgb_processing.model (the file size will be increased
> bit by bit) and dnn_rgb_processing.raw (the file size will not change much).
> > What about compress them to bz2? Uncompress them when need them?
> 
> No, we do not require bz2 for fate and should not add more requirements.

If there's no more comments, I'll keep one test for dnn_processing. thanks.

> 
> Carl Eugen
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Martin Storsjö Jan. 20, 2020, 1:24 p.m. UTC | #7
On Sat, 18 Jan 2020, Guo, Yejun wrote:

>
>
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
>> Carl Eugen Hoyos
>> Sent: Saturday, January 18, 2020 7:06 AM
>> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
>> check for dnn_processing
>>
>> Am Fr., 17. Jan. 2020 um 15:37 Uhr schrieb Guo, Yejun <yejun.guo@intel.com>:
>>>
>>> the reason is that the tested models are in float format
>>
>> The two reference files are ~1MB together: Do we want / need that?
>> (No opinion here.)
>
> how about just keeping one test (for format rgb24) for vf_dnn_processing? The rgb format might be the most popular format for vf_dnn_processing.
>
> And the test has two materials:
> 1)  fate-suite/dnn_processing/dnn_rgb_processing.model (can be renamed from halve_first_channel_float.model, 184 byte now)
> 2)  fate-suite/filter-reference/dnn_rgb_processing.raw (can be renamed from dnn_processing-halve_first_channel_float.raw, 446KB now)
>
> When we support more models (the most effort is to add more dnn layers in native mode),
> I can update both dnn_rgb_processing.model (the file size will be increased bit by bit) and dnn_rgb_processing.raw (the file size will not change much).

Keep in mind that ideally, you shouldn't be changing the reference files 
in the separate samples directory incrementially; ideally they should be 
fairly static.

Reducing the number of tests, if you need large test references, sounds 
sensible to me.

// Martin
Carl Eugen Hoyos Jan. 20, 2020, 2:14 p.m. UTC | #8
Am Mo., 20. Jan. 2020 um 14:25 Uhr schrieb Martin Storsjö <martin@martin.st>:

> Keep in mind that ideally, you shouldn't be changing the reference files
> in the separate samples directory incrementially; ideally they should be
> fairly static.

Since not everybody is a native speaker:
You cannot change reference files once they are used by fate, they have
to be static and remain where they are.

Carl Eugen
Guo, Yejun Jan. 20, 2020, 4:04 p.m. UTC | #9
> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> Carl Eugen Hoyos
> Sent: Monday, January 20, 2020 10:14 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> check for dnn_processing
> 
> Am Mo., 20. Jan. 2020 um 14:25 Uhr schrieb Martin Storsjö
> <martin@martin.st>:
> 
> > Keep in mind that ideally, you shouldn't be changing the reference files
> > in the separate samples directory incrementially; ideally they should be
> > fairly static.
> 
> Since not everybody is a native speaker:
> You cannot change reference files once they are used by fate, they have
> to be static and remain where they are.

thanks Carl.

Just had a chance to test on IBM PowerPC (big end) and found the new gray float test fails,
the reason is that the reference file is generated in little end machine and grayf32 contains 4 bytes.

So it makes sense to remove the test for grayf32 format, and only keep rgb24 format.

To make the reference file static, my idea is to keep the current fate-filter-dnn_processing-halve_first_channel_float (with reference file) unchanged, and only add a new test once we get a big milestone of dnn_processing.

> 
> Carl Eugen
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Martin Storsjö Jan. 20, 2020, 8:38 p.m. UTC | #10
On Mon, 20 Jan 2020, Guo, Yejun wrote:

>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
>> Carl Eugen Hoyos
>> Sent: Monday, January 20, 2020 10:14 PM
>> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
>> check for dnn_processing
>>
>> Am Mo., 20. Jan. 2020 um 14:25 Uhr schrieb Martin Storsjö
>> <martin@martin.st>:
>>
>>> Keep in mind that ideally, you shouldn't be changing the reference files
>>> in the separate samples directory incrementially; ideally they should be
>>> fairly static.
>>
>> Since not everybody is a native speaker:
>> You cannot change reference files once they are used by fate, they have
>> to be static and remain where they are.
>
> thanks Carl.
>
> Just had a chance to test on IBM PowerPC (big end) and found the new gray float test fails,
> the reason is that the reference file is generated in little end machine and grayf32 contains 4 bytes.

Just FWIW, for such cases, you should add something like "-pixfmt 
grayf32le", so that the output is independent of the host endianness and 
set e.g. CMP_UNIT=f32 and e.g. FUZZ=<value>, to make the oneoff test 
actually do the right thing, otherwise you'd just compare individual bytes 
in the float representation.

// Martin
Guo, Yejun Jan. 21, 2020, 3:08 a.m. UTC | #11
> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> Martin Storsj?
> Sent: Tuesday, January 21, 2020 4:38 AM
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> check for dnn_processing
> 
> On Mon, 20 Jan 2020, Guo, Yejun wrote:
> 
> >> -----Original Message-----
> >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> Of
> >> Carl Eugen Hoyos
> >> Sent: Monday, January 20, 2020 10:14 PM
> >> To: FFmpeg development discussions and patches
> <ffmpeg-devel@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use
> bit-exact
> >> check for dnn_processing
> >>
> >> Am Mo., 20. Jan. 2020 um 14:25 Uhr schrieb Martin Storsjö
> >> <martin@martin.st>:
> >>
> >>> Keep in mind that ideally, you shouldn't be changing the reference files
> >>> in the separate samples directory incrementially; ideally they should be
> >>> fairly static.
> >>
> >> Since not everybody is a native speaker:
> >> You cannot change reference files once they are used by fate, they have
> >> to be static and remain where they are.
> >
> > thanks Carl.
> >
> > Just had a chance to test on IBM PowerPC (big end) and found the new gray
> float test fails,
> > the reason is that the reference file is generated in little end machine and
> grayf32 contains 4 bytes.
> 
> Just FWIW, for such cases, you should add something like "-pixfmt
> grayf32le", so that the output is independent of the host endianness and
> set e.g. CMP_UNIT=f32 and e.g. FUZZ=<value>, to make the oneoff test
> actually do the right thing, otherwise you'd just compare individual bytes
> in the float representation.

thanks Martin, I understand it now.

so, just to make the reference files small, I'll remove the grayf32 test in V2 patch if no other comments, thanks.

> 
> // Martin
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Martin Storsjö Jan. 22, 2020, 12:09 p.m. UTC | #12
On Tue, 21 Jan 2020, Guo, Yejun wrote:

>
>
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
>> Martin Storsj?
>> Sent: Tuesday, January 21, 2020 4:38 AM
>> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
>> check for dnn_processing
>>
>> On Mon, 20 Jan 2020, Guo, Yejun wrote:
>>
>>>> -----Original Message-----
>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
>> Of
>>>> Carl Eugen Hoyos
>>>> Sent: Monday, January 20, 2020 10:14 PM
>>>> To: FFmpeg development discussions and patches
>> <ffmpeg-devel@ffmpeg.org>
>>>> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use
>> bit-exact
>>>> check for dnn_processing
>>>>
>>>> Am Mo., 20. Jan. 2020 um 14:25 Uhr schrieb Martin Storsjö
>>>> <martin@martin.st>:
>>>>
>>>>> Keep in mind that ideally, you shouldn't be changing the reference files
>>>>> in the separate samples directory incrementially; ideally they should be
>>>>> fairly static.
>>>>
>>>> Since not everybody is a native speaker:
>>>> You cannot change reference files once they are used by fate, they have
>>>> to be static and remain where they are.
>>>
>>> thanks Carl.
>>>
>>> Just had a chance to test on IBM PowerPC (big end) and found the new gray
>> float test fails,
>>> the reason is that the reference file is generated in little end machine and
>> grayf32 contains 4 bytes.
>>
>> Just FWIW, for such cases, you should add something like "-pixfmt
>> grayf32le", so that the output is independent of the host endianness and
>> set e.g. CMP_UNIT=f32 and e.g. FUZZ=<value>, to make the oneoff test
>> actually do the right thing, otherwise you'd just compare individual bytes
>> in the float representation.
>
> thanks Martin, I understand it now.
>
> so, just to make the reference files small, I'll remove the grayf32 test 
> in V2 patch if no other comments, thanks.

Removing one test sounds fine, and keeping one test with an external 
reference file in the samples directory sounds good to me - assuming that 
the reference output is stable and won't change in the forseeable future, 
and that you manage to get the test stable across architectures and 
different endianness.

If it takes time to get the test to that point, I would suggest reverting 
the existing two tests for now.

// Martin
Guo, Yejun Jan. 22, 2020, 1:34 p.m. UTC | #13
> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of
> Martin Storsj?
> Sent: Wednesday, January 22, 2020 8:09 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use bit-exact
> check for dnn_processing
> 
> On Tue, 21 Jan 2020, Guo, Yejun wrote:
> 
> >
> >
> >> -----Original Message-----
> >> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf
> Of
> >> Martin Storsj?
> >> Sent: Tuesday, January 21, 2020 4:38 AM
> >> To: FFmpeg development discussions and patches
> <ffmpeg-devel@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use
> bit-exact
> >> check for dnn_processing
> >>
> >> On Mon, 20 Jan 2020, Guo, Yejun wrote:
> >>
> >>>> -----Original Message-----
> >>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On
> Behalf
> >> Of
> >>>> Carl Eugen Hoyos
> >>>> Sent: Monday, January 20, 2020 10:14 PM
> >>>> To: FFmpeg development discussions and patches
> >> <ffmpeg-devel@ffmpeg.org>
> >>>> Subject: Re: [FFmpeg-devel] [PATCH] fate/filter-video.mak: do not use
> >> bit-exact
> >>>> check for dnn_processing
> >>>>
> >>>> Am Mo., 20. Jan. 2020 um 14:25 Uhr schrieb Martin Storsjö
> >>>> <martin@martin.st>:
> >>>>
> >>>>> Keep in mind that ideally, you shouldn't be changing the reference files
> >>>>> in the separate samples directory incrementially; ideally they should be
> >>>>> fairly static.
> >>>>
> >>>> Since not everybody is a native speaker:
> >>>> You cannot change reference files once they are used by fate, they have
> >>>> to be static and remain where they are.
> >>>
> >>> thanks Carl.
> >>>
> >>> Just had a chance to test on IBM PowerPC (big end) and found the new
> gray
> >> float test fails,
> >>> the reason is that the reference file is generated in little end machine and
> >> grayf32 contains 4 bytes.
> >>
> >> Just FWIW, for such cases, you should add something like "-pixfmt
> >> grayf32le", so that the output is independent of the host endianness and
> >> set e.g. CMP_UNIT=f32 and e.g. FUZZ=<value>, to make the oneoff test
> >> actually do the right thing, otherwise you'd just compare individual bytes
> >> in the float representation.
> >
> > thanks Martin, I understand it now.
> >
> > so, just to make the reference files small, I'll remove the grayf32 test
> > in V2 patch if no other comments, thanks.
> 
> Removing one test sounds fine, and keeping one test with an external
> reference file in the samples directory sounds good to me - assuming that
> the reference output is stable and won't change in the forseeable future,
> and that you manage to get the test stable across architectures and
> different endianness.

yes, I can just keep fate-filter-dnn_processing-halve_first_channel_float, and will
not change it in future.

The pixle format of the test is rgb24, there's no issue for endianess, and I've also
verified it on IBM PowerPC.

The only concern is the value of FUZZ for oneoff test, I think the default value 1
is enough for all the architectures, but I do not have one at hand for a real verification.

I'll send out the v2 patch, and heartily seek helps for a pre-push verification from whom have other systems except x86/linux. thanks.

> 
> If it takes time to get the test to that point, I would suggest reverting
> the existing two tests for now.
> 
> // Martin
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Carl Eugen Hoyos Jan. 29, 2020, 12:20 a.m. UTC | #14
Am Mi., 22. Jan. 2020 um 13:09 Uhr schrieb Martin Storsjö <martin@martin.st>:

> If it takes time to get the test to that point, I would suggest reverting
> the existing two tests for now.

Did this when I realized that the existing test breaks fate without
SAMPLES on all platforms.

Carl Eugen
Pedro Arthur Jan. 29, 2020, 3:40 a.m. UTC | #15
Em ter., 28 de jan. de 2020 às 21:20, Carl Eugen Hoyos
<ceffmpeg@gmail.com> escreveu:
>
> Am Mi., 22. Jan. 2020 um 13:09 Uhr schrieb Martin Storsjö <martin@martin.st>:
>
> > If it takes time to get the test to that point, I would suggest reverting
> > the existing two tests for now.
>
> Did this when I realized that the existing test breaks fate without
> SAMPLES on all platforms.
I'm not against reverting it but isn't SAMPLES required to run fate?
at least it seems to be implied by [1]

>
> Carl Eugen
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[1] - https://ffmpeg.org/fate.html
Carl Eugen Hoyos Jan. 30, 2020, 2:17 a.m. UTC | #16
Am Mi., 29. Jan. 2020 um 04:41 Uhr schrieb Pedro Arthur <bygrandao@gmail.com>:
>
> Em ter., 28 de jan. de 2020 às 21:20, Carl Eugen Hoyos
> <ceffmpeg@gmail.com> escreveu:
> >
> > Am Mi., 22. Jan. 2020 um 13:09 Uhr schrieb Martin Storsjö <martin@martin.st>:
> >
> > > If it takes time to get the test to that point, I would suggest reverting
> > > the existing two tests for now.
> >
> > Did this when I realized that the existing test breaks fate without
> > SAMPLES on all platforms.

> I'm not against reverting it but isn't SAMPLES required to run fate?

Definitely not?

Carl Eugen
diff mbox series

Patch

diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak
index 02986b5..dbf7994 100644
--- a/tests/fate/filter-video.mak
+++ b/tests/fate/filter-video.mak
@@ -260,8 +260,13 @@  FATE_FILTER_VSYNTH-$(CONFIG_PHASE_FILTER) += fate-filter-phase
 fate-filter-phase: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf phase
 
 FATE_FILTER_VSYNTH-$(CONFIG_DNN_PROCESSING_FILTER) += fate-filter-dnn_processing-halve_first_channel_float fate-filter-dnn_processing-halve_gray_float
-fate-filter-dnn_processing-halve_first_channel_float: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf format=rgb24,dnn_processing=model=$(TARGET_SAMPLES)/dnn_processing/halve_first_channel_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
-fate-filter-dnn_processing-halve_gray_float: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf format=grayf32,dnn_processing=model=$(TARGET_SAMPLES)/dnn_processing/halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native
+fate-filter-dnn_processing-halve_first_channel_float: CMD =  ffmpeg -i $(SRC) -frames:v 1 -vf format=rgb24,dnn_processing=model=$(TARGET_SAMPLES)/dnn_processing/halve_first_channel_float.model:input=dnn_in:output=dnn_out:dnn_backend=native -f rawvideo -
+fate-filter-dnn_processing-halve_first_channel_float: REF = $(SAMPLES)/filter-reference/dnn_processing-halve_first_channel_float.raw
+fate-filter-dnn_processing-halve_first_channel_float: CMP = oneoff
+
+fate-filter-dnn_processing-halve_gray_float: CMD = ffmpeg -i $(SRC) -frames:v 1 -vf format=grayf32,dnn_processing=model=$(TARGET_SAMPLES)/dnn_processing/halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native  -f rawvideo -
+fate-filter-dnn_processing-halve_gray_float: REF = $(SAMPLES)/filter-reference/dnn_processing-halve_gray_float.raw
+fate-filter-dnn_processing-halve_gray_float: CMP = oneoff
 
 FATE_REMOVEGRAIN += fate-filter-removegrain-mode-00
 fate-filter-removegrain-mode-00: CMD = framecrc -c:v pgmyuv -i $(SRC) -frames:v 1 -vf removegrain=0:0:0
diff --git a/tests/ref/fate/filter-dnn_processing-halve_first_channel_float b/tests/ref/fate/filter-dnn_processing-halve_first_channel_float
deleted file mode 100644
index ad7deda..0000000
--- a/tests/ref/fate/filter-dnn_processing-halve_first_channel_float
+++ /dev/null
@@ -1,55 +0,0 @@ 
-#tb 0: 1/25
-#media_type 0: video
-#codec_id 0: rawvideo
-#dimensions 0: 352x288
-#sar 0: 0/1
-0,          0,          0,        1,   304128, 0xdecb04c8
-0,          1,          1,        1,   304128, 0x55d9a543
-0,          2,          2,        1,   304128, 0x11ae175c
-0,          3,          3,        1,   304128, 0x2cbc8734
-0,          4,          4,        1,   304128, 0xa8e5cb1f
-0,          5,          5,        1,   304128, 0xb6d9ce81
-0,          6,          6,        1,   304128, 0x51613881
-0,          7,          7,        1,   304128, 0x09b9614f
-0,          8,          8,        1,   304128, 0x172a6901
-0,          9,          9,        1,   304128, 0x30237bf4
-0,         10,         10,        1,   304128, 0xb8646354
-0,         11,         11,        1,   304128, 0xe2dd8145
-0,         12,         12,        1,   304128, 0x6f0b3cea
-0,         13,         13,        1,   304128, 0xb1a04427
-0,         14,         14,        1,   304128, 0xe0ab42cf
-0,         15,         15,        1,   304128, 0xd4dc3224
-0,         16,         16,        1,   304128, 0xdb3462a1
-0,         17,         17,        1,   304128, 0x1d9931a1
-0,         18,         18,        1,   304128, 0x17c80e7a
-0,         19,         19,        1,   304128, 0x800d587b
-0,         20,         20,        1,   304128, 0x97d67832
-0,         21,         21,        1,   304128, 0xffc116db
-0,         22,         22,        1,   304128, 0x80510bc1
-0,         23,         23,        1,   304128, 0xbf838895
-0,         24,         24,        1,   304128, 0x3c8ce931
-0,         25,         25,        1,   304128, 0x8640e1cd
-0,         26,         26,        1,   304128, 0xa944fcac
-0,         27,         27,        1,   304128, 0x7cef3f83
-0,         28,         28,        1,   304128, 0x3c8d60d2
-0,         29,         29,        1,   304128, 0x83fad1ef
-0,         30,         30,        1,   304128, 0xbd6031ac
-0,         31,         31,        1,   304128, 0x9e63188a
-0,         32,         32,        1,   304128, 0x0e45cb70
-0,         33,         33,        1,   304128, 0x02a9ec32
-0,         34,         34,        1,   304128, 0x6ff674cc
-0,         35,         35,        1,   304128, 0x7d1143e6
-0,         36,         36,        1,   304128, 0x52c6b9b7
-0,         37,         37,        1,   304128, 0x16696d9c
-0,         38,         38,        1,   304128, 0x0612973f
-0,         39,         39,        1,   304128, 0xed130f6a
-0,         40,         40,        1,   304128, 0xe0051904
-0,         41,         41,        1,   304128, 0x6930d331
-0,         42,         42,        1,   304128, 0x35f722f7
-0,         43,         43,        1,   304128, 0x0adb7e81
-0,         44,         44,        1,   304128, 0x1eb10598
-0,         45,         45,        1,   304128, 0x73ec2115
-0,         46,         46,        1,   304128, 0xf9d24a8c
-0,         47,         47,        1,   304128, 0x94a3748d
-0,         48,         48,        1,   304128, 0xbaeac1d5
-0,         49,         49,        1,   304128, 0x5493efd3
diff --git a/tests/ref/fate/filter-dnn_processing-halve_gray_float b/tests/ref/fate/filter-dnn_processing-halve_gray_float
deleted file mode 100644
index b33a951..0000000
--- a/tests/ref/fate/filter-dnn_processing-halve_gray_float
+++ /dev/null
@@ -1,55 +0,0 @@ 
-#tb 0: 1/25
-#media_type 0: video
-#codec_id 0: rawvideo
-#dimensions 0: 352x288
-#sar 0: 0/1
-0,          0,          0,        1,   405504, 0xb3a2caab
-0,          1,          1,        1,   405504, 0x878b2fb8
-0,          2,          2,        1,   405504, 0xf39dac33
-0,          3,          3,        1,   405504, 0x94ef53b1
-0,          4,          4,        1,   405504, 0x6ed80f30
-0,          5,          5,        1,   405504, 0x82def1f8
-0,          6,          6,        1,   405504, 0xdafab027
-0,          7,          7,        1,   405504, 0xddef2774
-0,          8,          8,        1,   405504, 0x877771bd
-0,          9,          9,        1,   405504, 0xfaf7da12
-0,         10,         10,        1,   405504, 0x484be589
-0,         11,         11,        1,   405504, 0x15d660e2
-0,         12,         12,        1,   405504, 0xa01849c8
-0,         13,         13,        1,   405504, 0x823c33da
-0,         14,         14,        1,   405504, 0x3aef6445
-0,         15,         15,        1,   405504, 0x75f2961b
-0,         16,         16,        1,   405504, 0x7615ddac
-0,         17,         17,        1,   405504, 0xbcf92755
-0,         18,         18,        1,   405504, 0xc84ee75f
-0,         19,         19,        1,   405504, 0xf9d11220
-0,         20,         20,        1,   405504, 0x6e1afa4a
-0,         21,         21,        1,   405504, 0x47fcc8b2
-0,         22,         22,        1,   405504, 0xa5f618bc
-0,         23,         23,        1,   405504, 0x2528509b
-0,         24,         24,        1,   405504, 0x0b77ec0b
-0,         25,         25,        1,   405504, 0x8d5ea91d
-0,         26,         26,        1,   405504, 0xd8a04b22
-0,         27,         27,        1,   405504, 0xbde327bd
-0,         28,         28,        1,   405504, 0x9713aeb4
-0,         29,         29,        1,   405504, 0xc168c52e
-0,         30,         30,        1,   405504, 0xa3da9f70
-0,         31,         31,        1,   405504, 0xe58350d0
-0,         32,         32,        1,   405504, 0x6c656178
-0,         33,         33,        1,   405504, 0xe9563056
-0,         34,         34,        1,   405504, 0xf1f2c14d
-0,         35,         35,        1,   405504, 0x5d59fe20
-0,         36,         36,        1,   405504, 0x5ddb514e
-0,         37,         37,        1,   405504, 0x6251dbf8
-0,         38,         38,        1,   405504, 0x94c7d2d6
-0,         39,         39,        1,   405504, 0x1e44022a
-0,         40,         40,        1,   405504, 0x51c157a1
-0,         41,         41,        1,   405504, 0xc8991bd1
-0,         42,         42,        1,   405504, 0x046be642
-0,         43,         43,        1,   405504, 0x330da15f
-0,         44,         44,        1,   405504, 0xf6428e42
-0,         45,         45,        1,   405504, 0x8d303561
-0,         46,         46,        1,   405504, 0x135ed7d0
-0,         47,         47,        1,   405504, 0x0382f361
-0,         48,         48,        1,   405504, 0xddea2009
-0,         49,         49,        1,   405504, 0xd9b0262b