diff mbox series

[FFmpeg-devel,v2] fate: skip encoded output hashing for tests that depend on zlib

Message ID 20241019133358.9958-1-jamrial@gmail.com
State New
Headers show
Series [FFmpeg-devel,v2] fate: skip encoded output hashing for tests that depend on zlib | expand

Commit Message

James Almer Oct. 19, 2024, 1:33 p.m. UTC 
As there's no guarantee that the output will be exactly the same across implementations,
just don't test it at all.

Signed-off-by: James Almer <jamrial@gmail.com>
---
All tests covered now. Do we go this route, or update the zlib version of the
failing FATE machines?

 tests/fate-run.sh                         | 26 +++++++++++++++++------
 tests/fate/cover-art.mak                  |  6 +++---
 tests/fate/ffmpeg.mak                     |  2 +-
 tests/fate/image.mak                      |  4 ++--
 tests/fate/lavf-image.mak                 |  5 +++--
 tests/fate/lavf-video.mak                 |  4 ++--
 tests/fate/mov.mak                        |  2 +-
 tests/fate/vcodec.mak                     |  5 ++++-
 tests/ref/fate/copy-apng                  |  2 --
 tests/ref/fate/cover-art-aiff-id3v2-remux |  2 --
 tests/ref/fate/cover-art-flac-remux       |  2 --
 tests/ref/fate/cover-art-mp3-id3v2-remux  |  2 --
 tests/ref/fate/mov-cover-image            |  2 --
 tests/ref/fate/png-icc                    |  2 --
 tests/ref/fate/png-mdcv                   |  2 --
 tests/ref/lavf/apng                       |  2 --
 tests/ref/lavf/apng.png                   |  2 --
 tests/ref/lavf/gray16be.png               |  2 --
 tests/ref/lavf/png                        |  2 --
 tests/ref/lavf/rgb48be.png                |  2 --
 tests/ref/vsynth/vsynth1-flashsv          |  2 --
 tests/ref/vsynth/vsynth1-mpng             |  2 --
 tests/ref/vsynth/vsynth1-zlib             |  2 --
 tests/ref/vsynth/vsynth2-flashsv          |  2 --
 tests/ref/vsynth/vsynth2-mpng             |  2 --
 tests/ref/vsynth/vsynth2-zlib             |  2 --
 tests/ref/vsynth/vsynth3-flashsv          |  2 --
 tests/ref/vsynth/vsynth3-mpng             |  2 --
 tests/ref/vsynth/vsynth3-zlib             |  2 --
 tests/ref/vsynth/vsynth_lena-flashsv      |  2 --
 tests/ref/vsynth/vsynth_lena-mpng         |  2 --
 tests/ref/vsynth/vsynth_lena-zlib         |  2 --
 32 files changed, 36 insertions(+), 66 deletions(-)

Comments

Martin Storsjö Oct. 19, 2024, 2:04 p.m. UTC | #1 
On Sat, 19 Oct 2024, James Almer wrote:

> As there's no guarantee that the output will be exactly the same across implementations,
> just don't test it at all.
>
> Signed-off-by: James Almer <jamrial@gmail.com>
> ---
> All tests covered now. Do we go this route, or update the zlib version of the
> failing FATE machines?

I would say that requiring an upgrade isn't reasonsble here.

This isn't only about fate.ffmpeg.org, but also about any developer - and 
user compiling ffmpeg for themselves - to be able to verify that their 
build is good.

If we go down the "enforce upgrades" path, we pretty much force all users 
to upgrade or stop using zlib. For still supported, not entirely ancient 
distributions like Ubuntu 20.04, that seems a bit harsh for something 
which is mainly an issue with our tests.

// Martin
James Almer Oct. 19, 2024, 2:26 p.m. UTC | #2 
On 10/19/2024 11:04 AM, Martin Storsjö wrote:
> On Sat, 19 Oct 2024, James Almer wrote:
> 
>> As there's no guarantee that the output will be exactly the same 
>> across implementations,
>> just don't test it at all.
>>
>> Signed-off-by: James Almer <jamrial@gmail.com>
>> ---
>> All tests covered now. Do we go this route, or update the zlib version 
>> of the
>> failing FATE machines?
> 
> I would say that requiring an upgrade isn't reasonsble here.
> 
> This isn't only about fate.ffmpeg.org, but also about any developer - 
> and user compiling ffmpeg for themselves - to be able to verify that 
> their build is good.
> 
> If we go down the "enforce upgrades" path, we pretty much force all 
> users to upgrade or stop using zlib. For still supported, not entirely 
> ancient distributions like Ubuntu 20.04, that seems a bit harsh for 
> something which is mainly an issue with our tests.
But are those distros affected? Which zlib builds are faulty?
Martin Storsjö Oct. 19, 2024, 9:07 p.m. UTC | #3 
On Sat, 19 Oct 2024, James Almer wrote:

> On 10/19/2024 11:04 AM, Martin Storsjö wrote:
>> On Sat, 19 Oct 2024, James Almer wrote:
>> 
>>> As there's no guarantee that the output will be exactly the same 
>>> across implementations,
>>> just don't test it at all.
>>> 
>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>> ---
>>> All tests covered now. Do we go this route, or update the zlib version 
>>> of the
>>> failing FATE machines?
>> 
>> I would say that requiring an upgrade isn't reasonsble here.
>> 
>> This isn't only about fate.ffmpeg.org, but also about any developer - 
>> and user compiling ffmpeg for themselves - to be able to verify that 
>> their build is good.
>> 
>> If we go down the "enforce upgrades" path, we pretty much force all 
>> users to upgrade or stop using zlib. For still supported, not entirely 
>> ancient distributions like Ubuntu 20.04, that seems a bit harsh for 
>> something which is mainly an issue with our tests.
> But are those distros affected? Which zlib builds are faulty?

I don't remember exactly which versions that are affected (Ramiro pointed 
it out in some message), but at least Ubuntu 20.04 is affected.

// Martin
James Almer Oct. 19, 2024, 9:25 p.m. UTC | #4 
On 10/19/2024 6:07 PM, Martin Storsjö wrote:
> On Sat, 19 Oct 2024, James Almer wrote:
> 
>> On 10/19/2024 11:04 AM, Martin Storsjö wrote:
>>> On Sat, 19 Oct 2024, James Almer wrote:
>>>
>>>> As there's no guarantee that the output will be exactly the same 
>>>> across implementations,
>>>> just don't test it at all.
>>>>
>>>> Signed-off-by: James Almer <jamrial@gmail.com>
>>>> ---
>>>> All tests covered now. Do we go this route, or update the zlib 
>>>> version of the
>>>> failing FATE machines?
>>>
>>> I would say that requiring an upgrade isn't reasonsble here.
>>>
>>> This isn't only about fate.ffmpeg.org, but also about any developer - 
>>> and user compiling ffmpeg for themselves - to be able to verify that 
>>> their build is good.
>>>
>>> If we go down the "enforce upgrades" path, we pretty much force all 
>>> users to upgrade or stop using zlib. For still supported, not 
>>> entirely ancient distributions like Ubuntu 20.04, that seems a bit 
>>> harsh for something which is mainly an issue with our tests.
>> But are those distros affected? Which zlib builds are faulty?
> 
> I don't remember exactly which versions that are affected (Ramiro 
> pointed it out in some message), but at least Ubuntu 20.04 is affected.

Ok, if the zlib shipped by Ubuntu 20.04 is affected then yeah, we can't 
require a newer version until it goes EOL next April.
Michael Niedermayer Oct. 19, 2024, 10:09 p.m. UTC | #5 
On Sat, Oct 19, 2024 at 10:33:58AM -0300, James Almer wrote:
> As there's no guarantee that the output will be exactly the same across implementations,
> just don't test it at all.
> 
> Signed-off-by: James Almer <jamrial@gmail.com>

not objecting to this patch but

inflate is 550 lines of code
"Entire inflater (including optional zlib header parsing and Adler-32 checking) is implemented in a single function as a coroutine, which is separately available in a small (~550 line) source file: miniz_tinfl.c"

how much code do we need to write a simple store (not compress) ?
if i naively assume that decode must suport everything and encode
only one case that it chooses, iam perplexed how we still depend
on zlib for simply storing uncompressed zlib formated data

thx

[...]
James Almer Oct. 19, 2024, 10:22 p.m. UTC | #6 
On 10/19/2024 7:09 PM, Michael Niedermayer wrote:
> On Sat, Oct 19, 2024 at 10:33:58AM -0300, James Almer wrote:
>> As there's no guarantee that the output will be exactly the same across implementations,
>> just don't test it at all.
>>
>> Signed-off-by: James Almer <jamrial@gmail.com>
> 
> not objecting to this patch but
> 
> inflate is 550 lines of code
> "Entire inflater (including optional zlib header parsing and Adler-32 checking) is implemented in a single function as a coroutine, which is separately available in a small (~550 line) source file: miniz_tinfl.c"
> 
> how much code do we need to write a simple store (not compress) ?
> if i naively assume that decode must suport everything and encode
> only one case that it chooses, iam perplexed how we still depend
> on zlib for simply storing uncompressed zlib formated data

Well, nobody sets compression_level to 0 when encoding png or anything 
in any real world scenario, and this wasn't an issue until zlib-ng 
showed up.
But yes, a native uncompressed deflate() implementation was suggested, 
but nobody volunteered for it.
diff mbox series

Patch

diff --git a/tests/fate-run.sh b/tests/fate-run.sh
index 309ab85134..cf442daedf 100755
--- a/tests/fate-run.sh
+++ b/tests/fate-run.sh
@@ -217,6 +217,8 @@  enc_dec(){
     dec_opt_in=$7
     ffprobe_opts=$8
     twopass=$9
+    shift
+    no_file_checksums=$9
     encfile="${outdir}/${test}.${enc_fmt_out}"
     decfile="${outdir}/${test}.out.${dec_fmt_out}"
     cleanfiles="$cleanfiles $decfile"
@@ -237,8 +239,10 @@  enc_dec(){
 
     ffmpeg -auto_conversion_filters -f $enc_fmt_in $DEC_OPTS -i $tsrcfile $ENC_OPTS $enc_opt_out $FLAGS \
         -f $enc_fmt_out -y $tencfile || return
-    do_md5sum $encfile
-    echo $(wc -c $encfile)
+    if [ -z "$no_file_checksums" ]; then
+        do_md5sum $encfile
+        echo $(wc -c $encfile)
+    fi
     ffmpeg -auto_conversion_filters $dec_opt_in $DEC_OPTS -i $tencfile $ENC_OPTS $dec_opt_out $FLAGS \
         -f $dec_fmt_out -y $tdecfile || return
     do_md5sum $decfile
@@ -257,6 +261,8 @@  transcode(){
     additional_input=$7
     final_decode=$8
     enc_opt_in=$9
+    shift
+    no_file_checksums=$9
     test -z "$additional_input" || additional_input="$DEC_OPTS $additional_input"
     encfile="${outdir}/${test}.${enc_fmt}"
     test $keep -ge 1 || cleanfiles="$cleanfiles $encfile"
@@ -264,8 +270,10 @@  transcode(){
     tencfile=$(target_path $encfile)
     ffmpeg -f $src_fmt $DEC_OPTS $enc_opt_in -i $tsrcfile $additional_input \
            $ENC_OPTS $enc_opt $FLAGS -f $enc_fmt -y $tencfile || return
-    do_md5sum $encfile
-    echo $(wc -c $encfile)
+    if [ -z "$no_file_checksums" ]; then
+        do_md5sum $encfile
+        echo $(wc -c $encfile)
+    fi
     ffmpeg $DEC_OPTS $final_decode -i $tencfile $ENC_OPTS $FLAGS $final_encode \
         -f framecrc - || return
     test -z "$ffprobe_opts" || \
@@ -439,12 +447,18 @@  lavf_image2pipe(){
 }
 
 lavf_video(){
+    no_file_checksums="$3"
     t="${test#lavf-}"
     outdir="tests/data/lavf"
+    mkdir -p "$outdir"
     file=${outdir}/lavf.$t
     test "$keep" -ge 1 || cleanfiles="$cleanfiles $file"
-    do_avconv $file -auto_conversion_filters $DEC_OPTS -f image2 -c:v pgmyuv -i $raw_src \
-              "$ENC_OPTS -metadata title=lavftest" -t 1 -qscale 10 $1 $2 || return
+    run_avconv -auto_conversion_filters $DEC_OPTS -f image2 -c:v pgmyuv -i $raw_src \
+              "$ENC_OPTS -metadata title=lavftest" -t 1 -qscale 10 $1 $2 $target_path/$file || return
+    if [ -z "$no_file_checksums" ]; then
+        do_md5sum $file
+        echo $(wc -c $file)
+    fi
     do_avconv_crc $file -auto_conversion_filters $DEC_OPTS -i $target_path/$file $1
 }
 
diff --git a/tests/fate/cover-art.mak b/tests/fate/cover-art.mak
index faa2182745..8adfbf454c 100644
--- a/tests/fate/cover-art.mak
+++ b/tests/fate/cover-art.mak
@@ -37,14 +37,14 @@  FATE_COVER_ART_REMUX-$(call ALLYES, FILE_PROTOCOL FLAC_DEMUXER MJPEG_DECODER \
                                     AIFF_DEMUXER BMP_DECODER PNG_DECODER     \
                                     FRAMECRC_MUXER PIPE_PROTOCOL)            \
                        += fate-cover-art-aiff-id3v2-remux
-fate-cover-art-aiff-id3v2-remux: CMD = transcode flac $(TARGET_SAMPLES)/cover_art/cover_art.flac aiff "-map 0 -map 0:v -map 0:v -map 0:v -c:a pcm_s16be -c:v:0 copy -filter:v:1 scale -c:v:1 png -filter:v:2 scale -c:v:2 bmp -c:v:3 copy -write_id3v2 1 -metadata:g unknown_key=unknown_value -metadata compilation=foo -metadata:s:v:0 title=first -metadata:s:v:1 title=second -metadata:s:v:1 comment=Illustration -metadata:s:v:2 title=third -metadata:s:v:2 comment=Conductor -metadata:s:v:3 title=fourth -metadata:s:v:3 comment=Composer" "-map 0 -c copy -t 0.1" "-show_entries format_tags:stream_tags:stream_disposition=attached_pic:stream=index,codec_name"
+fate-cover-art-aiff-id3v2-remux: CMD = transcode flac $(TARGET_SAMPLES)/cover_art/cover_art.flac aiff "-map 0 -map 0:v -map 0:v -map 0:v -c:a pcm_s16be -c:v:0 copy -filter:v:1 scale -c:v:1 png -filter:v:2 scale -c:v:2 bmp -c:v:3 copy -write_id3v2 1 -metadata:g unknown_key=unknown_value -metadata compilation=foo -metadata:s:v:0 title=first -metadata:s:v:1 title=second -metadata:s:v:1 comment=Illustration -metadata:s:v:2 title=third -metadata:s:v:2 comment=Conductor -metadata:s:v:3 title=fourth -metadata:s:v:3 comment=Composer" "-map 0 -c copy -t 0.1" "-show_entries format_tags:stream_tags:stream_disposition=attached_pic:stream=index,codec_name" "" "" "" "no_file_checksums"
 
 FATE_COVER_ART_REMUX-$(call ALLYES, FILE_PROTOCOL MP3_DEMUXER MJPEG_DECODER \
                                     SCALE_FILTER PNG_ENCODER BMP_ENCODER    \
                                     MP3_MUXER BMP_DECODER PNG_DECODER       \
                                     FRAMECRC_MUXER PIPE_PROTOCOL)           \
                        += fate-cover-art-mp3-id3v2-remux
-fate-cover-art-mp3-id3v2-remux: CMD = transcode mp3 $(TARGET_SAMPLES)/exif/embedded_small.mp3 mp3 "-map 0 -map 0:v -map 0:v -c:a copy -filter:v:0 scale -filter:v:2 scale -c:v:0 bmp -c:v:1 copy -c:v:2 png -metadata:s:v:0 comment=Band/Orchestra" "-map 0 -c copy -t 0.1" "-show_entries stream_tags:stream_disposition=attached_pic:stream=index,codec_name"
+fate-cover-art-mp3-id3v2-remux: CMD = transcode mp3 $(TARGET_SAMPLES)/exif/embedded_small.mp3 mp3 "-map 0 -map 0:v -map 0:v -c:a copy -filter:v:0 scale -filter:v:2 scale -c:v:0 bmp -c:v:1 copy -c:v:2 png -metadata:s:v:0 comment=Band/Orchestra" "-map 0 -c copy -t 0.1" "-show_entries stream_tags:stream_disposition=attached_pic:stream=index,codec_name" "" "" "" "no_file_checksums"
 
 # Also covers muxing and demuxing of nonstandard channel layouts into FLAC
 # as well as the unorthodox multi_dim_quant option of the FLAC encoder.
@@ -55,7 +55,7 @@  FATE_COVER_ART_REMUX-$(call ALLYES, FILE_PROTOCOL MOV_DEMUXER OGG_DEMUXER   \
                                     FLAC_MUXER FLAC_DEMUXER FLAC_DECODER    \
                                     FRAMECRC_MUXER PIPE_PROTOCOL)           \
                        += fate-cover-art-flac-remux
-fate-cover-art-flac-remux: CMD = transcode mov $(TARGET_SAMPLES)/lossless-audio/inside.m4a flac "-map 0 -map 1:v -map 1:v -af channelmap=channel_layout=FL+FC,aresample -c:a flac -multi_dim_quant 1 -c:v:0 copy -metadata:s:v:0 comment=Illustration -metadata:s:v:0 title=OpenMusic  -filter:v:1 scale -c:v:1 png -metadata:s:v:1 title=landscape -c:v:2 copy -filter:v:3 scale -metadata:s:v:2 title=portrait -c:v:3 bmp  -metadata:s:v:3 comment=Conductor -c:v:4 copy -t 0.4" "-map 0 -map 0:a -c:a:0 copy -c:v copy" "-show_entries format_tags:stream_tags:stream_disposition=attached_pic:stream=index,codec_name" "-f ogg -i $(TARGET_SAMPLES)/cover_art/ogg_vorbiscomment_cover.opus"
+fate-cover-art-flac-remux: CMD = transcode mov $(TARGET_SAMPLES)/lossless-audio/inside.m4a flac "-map 0 -map 1:v -map 1:v -af channelmap=channel_layout=FL+FC,aresample -c:a flac -multi_dim_quant 1 -c:v:0 copy -metadata:s:v:0 comment=Illustration -metadata:s:v:0 title=OpenMusic  -filter:v:1 scale -c:v:1 png -metadata:s:v:1 title=landscape -c:v:2 copy -filter:v:3 scale -metadata:s:v:2 title=portrait -c:v:3 bmp  -metadata:s:v:3 comment=Conductor -c:v:4 copy -t 0.4" "-map 0 -map 0:a -c:a:0 copy -c:v copy" "-show_entries format_tags:stream_tags:stream_disposition=attached_pic:stream=index,codec_name" "-f ogg -i $(TARGET_SAMPLES)/cover_art/ogg_vorbiscomment_cover.opus" "" "" "no_file_checksums"
 
 FCA_TEMP-$(call ALLYES, RAWVIDEO_MUXER FILE_PROTOCOL) = $(FATE_COVER_ART-yes)
 FATE_COVER_ART = $(FCA_TEMP-yes)
diff --git a/tests/fate/ffmpeg.mak b/tests/fate/ffmpeg.mak
index 869376dd18..96728f96c0 100644
--- a/tests/fate/ffmpeg.mak
+++ b/tests/fate/ffmpeg.mak
@@ -161,7 +161,7 @@  FATE_STREAMCOPY-$(call TRANSCODE, RAWVIDEO APNG, APNG) += fate-copy-apng
 endif
 fate-copy-apng: fate-lavf-apng
 fate-lavf-apng: KEEP_FILES ?= 1
-fate-copy-apng: CMD = transcode apng tests/data/lavf/lavf.apng apng "-c:v copy"
+fate-copy-apng: CMD = transcode apng tests/data/lavf/lavf.apng apng "-c:v copy" "" "" "" "" "" "no_file_checksums"
 
 FATE_STREAMCOPY-$(call DEMMUX, OGG, OGG) += fate-limited_input_seek fate-limited_input_seek-copyts
 fate-limited_input_seek: CMD = md5 -ss 1.5 -t 1.3 -i $(TARGET_SAMPLES)/vorbis/moog_small.ogg -c:a copy -fflags +bitexact -f ogg
diff --git a/tests/fate/image.mak b/tests/fate/image.mak
index 042cf6438f..11bb058826 100644
--- a/tests/fate/image.mak
+++ b/tests/fate/image.mak
@@ -410,7 +410,7 @@  fate-png-side-data: CMD = run ffprobe$(PROGSSUF)$(EXESUF) -show_frames \
     -i $(TARGET_SAMPLES)/png1/lena-int_rgb24.png
 
 FATE_PNG_TRANSCODE-$(call TRANSCODE, PNG, IMAGE2 IMAGE_PNG_PIPE) += fate-png-icc
-fate-png-icc: CMD = transcode png_pipe $(TARGET_SAMPLES)/png1/lena-int_rgb24.png image2 "-c png" "" "-show_frames"
+fate-png-icc: CMD = transcode png_pipe $(TARGET_SAMPLES)/png1/lena-int_rgb24.png image2 "-c png" "" "-show_frames" "" "" "" "no_file_checksums"
 
 FATE_PNG_PROBE-$(call ALLYES, LCMS2) += fate-png-icc-parse
 fate-png-icc-parse: CMD = run ffprobe$(PROGSSUF)$(EXESUF) -show_frames \
@@ -420,7 +420,7 @@  FATE_PNG_TRANSCODE-$(call TRANSCODE, PNG HEVC, IMAGE2PIPE HEVC, \
     IMAGE_PNG_PIPE_DEMUXER HEVC_PARSER PNG_DECODER SCALE_FILTER) += fate-png-mdcv
 fate-png-mdcv: CMD = transcode hevc $(TARGET_SAMPLES)/hevc/hdr10_plus_h265_sample.hevc image2pipe \
     "-pix_fmt rgb24 -vf scale -c png" "" \
-    "-show_frames -show_entries frame=side_data_list -of flat"
+    "-show_frames -show_entries frame=side_data_list -of flat" "" "" "" "no_file_checksums"
 
 FATE_PNG-$(call DEMDEC, IMAGE2, PNG) += $(FATE_PNG)
 FATE_PNG_PROBE-$(call DEMDEC, IMAGE2, PNG) += $(FATE_PNG_PROBE)
diff --git a/tests/fate/lavf-image.mak b/tests/fate/lavf-image.mak
index 4177e091b3..236bbcf513 100644
--- a/tests/fate/lavf-image.mak
+++ b/tests/fate/lavf-image.mak
@@ -88,8 +88,9 @@  fate-lavf-grayf32le.pfm:   CMD = lavf_image "-pix_fmt grayf32le" "-pix_fmt grayf
 fate-lavf-grayf32be.pfm:   CMD = lavf_image "-pix_fmt grayf32be" "-pix_fmt grayf32be"
 fate-lavf-gbrpf32le.pfm:   CMD = lavf_image "-pix_fmt gbrpf32le" "-pix_fmt gbrpf32le"
 fate-lavf-gbrpf32be.pfm:   CMD = lavf_image "-pix_fmt gbrpf32be" "-pix_fmt gbrpf32be"
-fate-lavf-gray16be.png: CMD = lavf_image "-pix_fmt gray16be"
-fate-lavf-rgb48be.png: CMD = lavf_image "-pix_fmt rgb48be"
+fate-lavf-png: CMD = lavf_image "-pix_fmt rgb24" "" "no_file_checksums"
+fate-lavf-gray16be.png: CMD = lavf_image "-pix_fmt gray16be" "" "no_file_checksums"
+fate-lavf-rgb48be.png: CMD = lavf_image "-pix_fmt rgb48be" "" "no_file_checksums"
 fate-lavf-rgba.xwd: CMD = lavf_image "-pix_fmt rgba"
 fate-lavf-rgb565be.xwd: CMD = lavf_image "-pix_fmt rgb565be"
 fate-lavf-rgb555be.xwd: CMD = lavf_image "-pix_fmt rgb555be"
diff --git a/tests/fate/lavf-video.mak b/tests/fate/lavf-video.mak
index da3b114bc8..b7970a9e1f 100644
--- a/tests/fate/lavf-video.mak
+++ b/tests/fate/lavf-video.mak
@@ -19,8 +19,8 @@  $(FATE_LAVF_VIDEO): CMD = lavf_video
 $(FATE_LAVF_VIDEO): REF = $(SRC_PATH)/tests/ref/lavf/$(@:fate-lavf-%=%)
 $(FATE_LAVF_VIDEO): $(VREF)
 
-fate-lavf-apng: CMD = lavf_video "-pix_fmt rgb24"
-fate-lavf-apng.png: CMD = lavf_video "-pix_fmt rgb24" "-frames:v 1 -f apng"
+fate-lavf-apng: CMD = lavf_video "-pix_fmt rgb24" "" "no_file_checksums"
+fate-lavf-apng.png: CMD = lavf_video "-pix_fmt rgb24" "-frames:v 1 -f apng" "no_file_checksums"
 fate-lavf-gray.fits: CMD = lavf_video "-pix_fmt gray"
 fate-lavf-gray16be.fits: CMD = lavf_video "-pix_fmt gray16be"
 fate-lavf-gbrp.fits: CMD = lavf_video "-pix_fmt gbrp"
diff --git a/tests/fate/mov.mak b/tests/fate/mov.mak
index ca13ebfd44..845d577bcf 100644
--- a/tests/fate/mov.mak
+++ b/tests/fate/mov.mak
@@ -138,7 +138,7 @@  fate-mov-mp4-chapters: CMD = transcode ogg $(TARGET_SAMPLES)/vorbis/vorbis_chapt
 
 FATE_MOV_FFMPEG_FFPROBE_SAMPLES-$(call TRANSCODE, PNG, MP4 MOV, MJPEG_DECODER SCALE_FILTER) \
                           += fate-mov-cover-image
-fate-mov-cover-image: CMD = transcode mov $(TARGET_SAMPLES)/cover_art/Owner-iTunes_9.0.3.15.m4a mp4 "-map 0 -map 0:v -c:a copy -c:v:0 copy -filter:v:1 scale -c:v:1 png" "-map 0 -t 0.1 -c copy" "-show_entries stream_disposition=attached_pic:stream=index,codec_name"
+fate-mov-cover-image: CMD = transcode mov $(TARGET_SAMPLES)/cover_art/Owner-iTunes_9.0.3.15.m4a mp4 "-map 0 -map 0:v -c:a copy -c:v:0 copy -filter:v:1 scale -c:v:1 png" "-map 0 -t 0.1 -c copy" "-show_entries stream_disposition=attached_pic:stream=index,codec_name" "" "" "" "no_file_checksums"
 
 FATE_MOV_FFMPEG_FFPROBE_SAMPLES-$(call TRANSCODE, TTML SUBRIP, MP4 MOV, SRT_DEMUXER TTML_MUXER) += fate-mov-mp4-ttml-stpp fate-mov-mp4-ttml-dfxp
 fate-mov-mp4-ttml-stpp: CMD = transcode srt $(TARGET_SAMPLES)/sub/SubRip_capability_tester.srt mp4 "-map 0:s -c:s ttml -time_base:s 1:1000" "-map 0 -c copy" "-of json -show_entries packet:stream=index,codec_type,codec_tag_string,codec_tag,codec_name,time_base,start_time,duration_ts,duration,nb_frames,nb_read_packets:stream_tags"
diff --git a/tests/fate/vcodec.mak b/tests/fate/vcodec.mak
index a12eb43f22..20bc2fc340 100644
--- a/tests/fate/vcodec.mak
+++ b/tests/fate/vcodec.mak
@@ -6,7 +6,7 @@  fate-vsynth%: CODEC = $(word 3, $(subst -, ,$(@)))
 fate-vsynth%: FMT = avi
 fate-vsynth%: DEFAULT_SIZE = -s 352x288
 fate-vsynth3-%: DEFAULT_SIZE = -s $(FATEW)x$(FATEH)
-fate-vsynth%: CMD = enc_dec "rawvideo $(DEFAULT_SIZE) -color_range mpeg -pix_fmt yuv420p $(RAWDECOPTS)" $(SRC) $(FMT) "-c $(CODEC) $(ENCOPTS)" rawvideo "-pix_fmt yuv420p -color_range mpeg -fps_mode passthrough $(DECOPTS)" "" "" ${TWOPASS}
+fate-vsynth%: CMD = enc_dec "rawvideo $(DEFAULT_SIZE) -color_range mpeg -pix_fmt yuv420p $(RAWDECOPTS)" $(SRC) $(FMT) "-c $(CODEC) $(ENCOPTS)" rawvideo "-pix_fmt yuv420p -color_range mpeg -fps_mode passthrough $(DECOPTS)" "" "" "${TWOPASS}" "${NO_FILE_CHECKSUMS}"
 fate-vsynth%: CMP_UNIT = 1
 fate-vsynth%: REF = $(SRC_PATH)/tests/ref/vsynth/$(@:fate-%=%)
 
@@ -193,6 +193,7 @@  FATE_VCODEC_SCALE-$(call ENCDEC, FLASHSV, FLV) += flashsv
 fate-vsynth%-flashsv:            ENCOPTS = -sws_flags neighbor+full_chroma_int
 fate-vsynth%-flashsv:            DECOPTS = -sws_flags area
 fate-vsynth%-flashsv:            FMT     = flv
+fate-vsynth%-flashsv:            NO_FILE_CHECKSUMS = 1
 
 FATE_VCODEC_SCALE-$(call ENCDEC, FLASHSV2, FLV) += flashsv2
 fate-vsynth%-flashsv2:           ENCOPTS = -sws_flags neighbor+full_chroma_int -strict experimental -compression_level 0
@@ -348,6 +349,7 @@  fate-vsynth%-msrle:              CODEC   = msrle
 
 FATE_VCODEC_SCALE-$(call ENCDEC, PNG, AVI) += mpng
 fate-vsynth%-mpng:               CODEC   = png
+fate-vsynth%-mpng:               NO_FILE_CHECKSUMS = 1
 
 FATE_VCODEC_SCALE-$(call ENCDEC, MSVIDEO1, AVI) += msvideo1
 
@@ -486,6 +488,7 @@  FATE_VCODEC-$(call ENCDEC, YUV4, AVI) += yuv4
 FATE_VCODEC_SCALE-$(call ENCDEC, Y41P, AVI) += y41p
 
 FATE_VCODEC_SCALE-$(call ENCDEC, ZLIB, AVI) += zlib
+fate-vsynth%-zlib:               NO_FILE_CHECKSUMS = 1
 
 FATE_VCODEC-$(CONFIG_SCALE_FILTER) += $(FATE_VCODEC_SCALE-yes)
 FATE_VCODEC += $(FATE_VCODEC-yes)
diff --git a/tests/ref/fate/copy-apng b/tests/ref/fate/copy-apng
index 3057573cee..17c9f1fdc5 100644
--- a/tests/ref/fate/copy-apng
+++ b/tests/ref/fate/copy-apng
@@ -1,5 +1,3 @@ 
-a4c46fad7716ad094eb3c78b74ca0244 *tests/data/fate/copy-apng.apng
-6209864 tests/data/fate/copy-apng.apng
 #tb 0: 1/25
 #media_type 0: video
 #codec_id 0: rawvideo
diff --git a/tests/ref/fate/cover-art-aiff-id3v2-remux b/tests/ref/fate/cover-art-aiff-id3v2-remux
index a59ba37c65..a938ff711a 100644
--- a/tests/ref/fate/cover-art-aiff-id3v2-remux
+++ b/tests/ref/fate/cover-art-aiff-id3v2-remux
@@ -1,5 +1,3 @@ 
-330ad2bf538e91a31752b38024461df1 *tests/data/fate/cover-art-aiff-id3v2-remux.aiff
-608914 tests/data/fate/cover-art-aiff-id3v2-remux.aiff
 #tb 0: 1/44100
 #media_type 0: audio
 #codec_id 0: pcm_s16be
diff --git a/tests/ref/fate/cover-art-flac-remux b/tests/ref/fate/cover-art-flac-remux
index fa91975881..c8dd74e7e1 100644
--- a/tests/ref/fate/cover-art-flac-remux
+++ b/tests/ref/fate/cover-art-flac-remux
@@ -1,5 +1,3 @@ 
-6defc5081a59ab12c8a5f9e263b25068 *tests/data/fate/cover-art-flac-remux.flac
-1098537 tests/data/fate/cover-art-flac-remux.flac
 #extradata 0:       34, 0x8d830abd
 #tb 0: 1/44100
 #media_type 0: audio
diff --git a/tests/ref/fate/cover-art-mp3-id3v2-remux b/tests/ref/fate/cover-art-mp3-id3v2-remux
index 52b7e72a56..bb54704fe8 100644
--- a/tests/ref/fate/cover-art-mp3-id3v2-remux
+++ b/tests/ref/fate/cover-art-mp3-id3v2-remux
@@ -1,5 +1,3 @@ 
-94946f0efd5f9bb0061ac1fbff7d731f *tests/data/fate/cover-art-mp3-id3v2-remux.mp3
-399346 tests/data/fate/cover-art-mp3-id3v2-remux.mp3
 #tb 0: 1/14112000
 #media_type 0: audio
 #codec_id 0: mp3
diff --git a/tests/ref/fate/mov-cover-image b/tests/ref/fate/mov-cover-image
index 5f65c630ea..6348df073a 100644
--- a/tests/ref/fate/mov-cover-image
+++ b/tests/ref/fate/mov-cover-image
@@ -1,5 +1,3 @@ 
-54a8870d5d1e6cc4da28ae422aa70898 *tests/data/fate/mov-cover-image.mp4
-1011919 tests/data/fate/mov-cover-image.mp4
 #extradata 0:        2, 0x00340022
 #tb 0: 1/44100
 #media_type 0: audio
diff --git a/tests/ref/fate/png-icc b/tests/ref/fate/png-icc
index 19f8c0a752..ddd5502bec 100644
--- a/tests/ref/fate/png-icc
+++ b/tests/ref/fate/png-icc
@@ -1,5 +1,3 @@ 
-c460cd06a88ace94d7c76a6309aa8fb8 *tests/data/fate/png-icc.image2
-49441 tests/data/fate/png-icc.image2
 #tb 0: 1/25
 #media_type 0: video
 #codec_id 0: rawvideo
diff --git a/tests/ref/fate/png-mdcv b/tests/ref/fate/png-mdcv
index c524a94ded..eb88c53601 100644
--- a/tests/ref/fate/png-mdcv
+++ b/tests/ref/fate/png-mdcv
@@ -1,5 +1,3 @@ 
-fc68fe6c8c72343b96d2695f6913995b *tests/data/fate/png-mdcv.image2pipe
-439248 tests/data/fate/png-mdcv.image2pipe
 #tb 0: 1/25
 #media_type 0: video
 #codec_id 0: rawvideo
diff --git a/tests/ref/lavf/apng b/tests/ref/lavf/apng
index 95f1fbee6f..b52460b8af 100644
--- a/tests/ref/lavf/apng
+++ b/tests/ref/lavf/apng
@@ -1,3 +1 @@ 
-a4c46fad7716ad094eb3c78b74ca0244 *tests/data/lavf/lavf.apng
-6209864 tests/data/lavf/lavf.apng
 tests/data/lavf/lavf.apng CRC=0x87b3c15f
diff --git a/tests/ref/lavf/apng.png b/tests/ref/lavf/apng.png
index b42f5a8db8..f23f6ccb23 100644
--- a/tests/ref/lavf/apng.png
+++ b/tests/ref/lavf/apng.png
@@ -1,3 +1 @@ 
-c5900fdd1b2fc30b985793f5226fd0c4 *tests/data/lavf/lavf.apng.png
-248854 tests/data/lavf/lavf.apng.png
 tests/data/lavf/lavf.apng.png CRC=0xd8c7b7a1
diff --git a/tests/ref/lavf/gray16be.png b/tests/ref/lavf/gray16be.png
index 4f4ce179d2..cd759993c1 100644
--- a/tests/ref/lavf/gray16be.png
+++ b/tests/ref/lavf/gray16be.png
@@ -1,3 +1 @@ 
-6cf54c13aa407b77547cf6dfe23ecba3 *tests/data/images/gray16be.png/02.gray16be.png
-47365 tests/data/images/gray16be.png/02.gray16be.png
 tests/data/images/gray16be.png/%02d.gray16be.png CRC=0x893f10ef
diff --git a/tests/ref/lavf/png b/tests/ref/lavf/png
index 2385ad1800..3ef1219492 100644
--- a/tests/ref/lavf/png
+++ b/tests/ref/lavf/png
@@ -1,3 +1 @@ 
-2af72da4468e61a37c220b25cb28618a *tests/data/images/png/02.png
-248633 tests/data/images/png/02.png
 tests/data/images/png/%02d.png CRC=0x6da01946
diff --git a/tests/ref/lavf/rgb48be.png b/tests/ref/lavf/rgb48be.png
index b893401a8f..9e1e50c2ee 100644
--- a/tests/ref/lavf/rgb48be.png
+++ b/tests/ref/lavf/rgb48be.png
@@ -1,3 +1 @@ 
-b4e38244c97debe3f528e7d1adb283ef *tests/data/images/rgb48be.png/02.rgb48be.png
-511900 tests/data/images/rgb48be.png/02.rgb48be.png
 tests/data/images/rgb48be.png/%02d.rgb48be.png CRC=0x5984c023
diff --git a/tests/ref/vsynth/vsynth1-flashsv b/tests/ref/vsynth/vsynth1-flashsv
index 05f9e9432d..727697c5ab 100644
--- a/tests/ref/vsynth/vsynth1-flashsv
+++ b/tests/ref/vsynth/vsynth1-flashsv
@@ -1,4 +1,2 @@ 
-61b0825258ac6fe85691bdefd892960d *tests/data/fate/vsynth1-flashsv.flv
-14681925 tests/data/fate/vsynth1-flashsv.flv
 791e1fb999deb2e4156e2286d48c4ed1 *tests/data/fate/vsynth1-flashsv.out.rawvideo
 stddev:    2.84 PSNR: 39.04 MAXDIFF:   49 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth1-mpng b/tests/ref/vsynth/vsynth1-mpng
index 9b394d7629..8547ef1c05 100644
--- a/tests/ref/vsynth/vsynth1-mpng
+++ b/tests/ref/vsynth/vsynth1-mpng
@@ -1,4 +1,2 @@ 
-6a27410a07ed1c5556e15b7a7c6a586d *tests/data/fate/vsynth1-mpng.avi
-12158280 tests/data/fate/vsynth1-mpng.avi
 93695a27c24a61105076ca7b1f010bbd *tests/data/fate/vsynth1-mpng.out.rawvideo
 stddev:    3.42 PSNR: 37.44 MAXDIFF:   48 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth1-zlib b/tests/ref/vsynth/vsynth1-zlib
index 419c6a6b3a..68b314a4a2 100644
--- a/tests/ref/vsynth/vsynth1-zlib
+++ b/tests/ref/vsynth/vsynth1-zlib
@@ -1,4 +1,2 @@ 
-fd44e7bf6322f4be46f4f0dada728849 *tests/data/fate/vsynth1-zlib.avi
-12108636 tests/data/fate/vsynth1-zlib.avi
 93695a27c24a61105076ca7b1f010bbd *tests/data/fate/vsynth1-zlib.out.rawvideo
 stddev:    3.42 PSNR: 37.44 MAXDIFF:   48 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth2-flashsv b/tests/ref/vsynth/vsynth2-flashsv
index f659edaab5..36c9799472 100644
--- a/tests/ref/vsynth/vsynth2-flashsv
+++ b/tests/ref/vsynth/vsynth2-flashsv
@@ -1,4 +1,2 @@ 
-a2f145e6e44b51f8fc64ead06a994273 *tests/data/fate/vsynth2-flashsv.flv
-11637702 tests/data/fate/vsynth2-flashsv.flv
 7f0fc12c02e68faddc153e69ddd6841c *tests/data/fate/vsynth2-flashsv.out.rawvideo
 stddev:    1.20 PSNR: 46.52 MAXDIFF:   20 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth2-mpng b/tests/ref/vsynth/vsynth2-mpng
index e49714a3b0..a811778bce 100644
--- a/tests/ref/vsynth/vsynth2-mpng
+++ b/tests/ref/vsynth/vsynth2-mpng
@@ -1,4 +1,2 @@ 
-481e2d148f411bb61783aa688ec22943 *tests/data/fate/vsynth2-mpng.avi
-11816978 tests/data/fate/vsynth2-mpng.avi
 32fae3e665407bb4317b3f90fedb903c *tests/data/fate/vsynth2-mpng.out.rawvideo
 stddev:    1.54 PSNR: 44.37 MAXDIFF:   17 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth2-zlib b/tests/ref/vsynth/vsynth2-zlib
index 9dfb0d679e..96bb27005e 100644
--- a/tests/ref/vsynth/vsynth2-zlib
+++ b/tests/ref/vsynth/vsynth2-zlib
@@ -1,4 +1,2 @@ 
-8d2783483b354e7836cfc45a7241c4eb *tests/data/fate/vsynth2-zlib.avi
-11760520 tests/data/fate/vsynth2-zlib.avi
 32fae3e665407bb4317b3f90fedb903c *tests/data/fate/vsynth2-zlib.out.rawvideo
 stddev:    1.54 PSNR: 44.37 MAXDIFF:   17 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth3-flashsv b/tests/ref/vsynth/vsynth3-flashsv
index cdf9e7bf99..938a9cc906 100644
--- a/tests/ref/vsynth/vsynth3-flashsv
+++ b/tests/ref/vsynth/vsynth3-flashsv
@@ -1,4 +1,2 @@ 
-a05a9ab0ae21925dd10a83639de77d77 *tests/data/fate/vsynth3-flashsv.flv
-171419 tests/data/fate/vsynth3-flashsv.flv
 faa660b0ecaaab1bf9b5d7284019aa01 *tests/data/fate/vsynth3-flashsv.out.rawvideo
 stddev:    2.97 PSNR: 38.67 MAXDIFF:   49 bytes:    86700/    86700
diff --git a/tests/ref/vsynth/vsynth3-mpng b/tests/ref/vsynth/vsynth3-mpng
index e708a60d98..75fa033737 100644
--- a/tests/ref/vsynth/vsynth3-mpng
+++ b/tests/ref/vsynth/vsynth3-mpng
@@ -1,4 +1,2 @@ 
-3f64b66a1f46e31d45dd7f5514422ed0 *tests/data/fate/vsynth3-mpng.avi
-179804 tests/data/fate/vsynth3-mpng.avi
 693aff10c094f8bd31693f74cf79d2b2 *tests/data/fate/vsynth3-mpng.out.rawvideo
 stddev:    3.67 PSNR: 36.82 MAXDIFF:   43 bytes:    86700/    86700
diff --git a/tests/ref/vsynth/vsynth3-zlib b/tests/ref/vsynth/vsynth3-zlib
index 55d65f30e0..3012be1053 100644
--- a/tests/ref/vsynth/vsynth3-zlib
+++ b/tests/ref/vsynth/vsynth3-zlib
@@ -1,4 +1,2 @@ 
-d3705acfc90b16968d0d4a3ac56b049c *tests/data/fate/vsynth3-zlib.avi
-174882 tests/data/fate/vsynth3-zlib.avi
 693aff10c094f8bd31693f74cf79d2b2 *tests/data/fate/vsynth3-zlib.out.rawvideo
 stddev:    3.67 PSNR: 36.82 MAXDIFF:   43 bytes:    86700/    86700
diff --git a/tests/ref/vsynth/vsynth_lena-flashsv b/tests/ref/vsynth/vsynth_lena-flashsv
index 7dfec3edf5..a5ebe87783 100644
--- a/tests/ref/vsynth/vsynth_lena-flashsv
+++ b/tests/ref/vsynth/vsynth_lena-flashsv
@@ -1,4 +1,2 @@ 
-59cdcf5f92c5113e27b2fda632ccd55c *tests/data/fate/vsynth_lena-flashsv.flv
-12368953 tests/data/fate/vsynth_lena-flashsv.flv
 3a984506f1ebfc9fb73b6814cab201cc *tests/data/fate/vsynth_lena-flashsv.out.rawvideo
 stddev:    0.66 PSNR: 51.73 MAXDIFF:   14 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth_lena-mpng b/tests/ref/vsynth/vsynth_lena-mpng
index 20258433b0..11e84a227b 100644
--- a/tests/ref/vsynth/vsynth_lena-mpng
+++ b/tests/ref/vsynth/vsynth_lena-mpng
@@ -1,4 +1,2 @@ 
-319ca02f8465a9c53128442f4b0dd528 *tests/data/fate/vsynth_lena-mpng.avi
-12558334 tests/data/fate/vsynth_lena-mpng.avi
 98d0e2854731472c5bf13d8638502d0a *tests/data/fate/vsynth_lena-mpng.out.rawvideo
 stddev:    1.26 PSNR: 46.10 MAXDIFF:   13 bytes:  7603200/  7603200
diff --git a/tests/ref/vsynth/vsynth_lena-zlib b/tests/ref/vsynth/vsynth_lena-zlib
index 009bcad853..aaed71cc40 100644
--- a/tests/ref/vsynth/vsynth_lena-zlib
+++ b/tests/ref/vsynth/vsynth_lena-zlib
@@ -1,4 +1,2 @@ 
-abbfccc551b173a357024d8392de21dd *tests/data/fate/vsynth_lena-zlib.avi
-12517180 tests/data/fate/vsynth_lena-zlib.avi
 98d0e2854731472c5bf13d8638502d0a *tests/data/fate/vsynth_lena-zlib.out.rawvideo
 stddev:    1.26 PSNR: 46.10 MAXDIFF:   13 bytes:  7603200/  7603200