Message ID | 378315057.2531045.1535207584068@mail.yahoo.com |
---|---|
State | Superseded |
Headers | show |
Please use git send-email to send your patches, or at least send each patch, created by git format-patch, as individual attachment. Your files seem to contain multiple patches one after another, which makes them very hard to follow. But nice work! Let's hope this does not cause any regressions.
On Sat, Aug 25, 2018 at 02:33:03PM +0000, avih wrote: > Hi, > > I noticed that configure can be a bit slow - few minutes on my macOS > system, 1:30m on linux/bash, 30s on linux/dash, and even if > pathological, still a too-long 10 minutes on Windows (MSYS2) (same > duration as a full build with make -j4 after configure). > > I added some timing information printouts for various parts in > configure, and eventually identified few culprits. > > The attachment "config-timing.patch" adds these printouts and sorted > summary. It's attached for reference and not intended to be merged. > It applies cleanly before or after the main patch. > > > The attached "main.patch" addresses three areas I identified as slow: > > 1. About 50-70% of configure runtime was being spent inside one > function: flatten_extralibs() and callees resolve() and unique(). > It manipulates strings and invoked nearly 20K (20000) subshells. > It was rewritten to avoid subshells, and ended up x50-x250 faster. > > 2. print_enabled_components() was invoking sed about 350 times on one > file. This is never instant but takes many seconds where fork is > slow (Windows). Invoke sed only once instead = x4-x10 speedup. > > 3. After the previous speedups, configure spent 20-60% of its runtime > at check_deps(). It's particularly slow with bash. After some local > optimizations - mainly avoid pushvar/popvar and abort early in one > notable case (empty deps), it's now x4-x25 faster. > > > Some resulting speedups (more details at "before-after.txt"): > > - macOS/bash: total: 98s -> 22s flatten_extralibs: 53s -> 0.7s > - Linux/bash: total: 87s -> 11s flatten_extralibs: 59s -> 0.9s > - Linux/dash: total: 27s -> 8s flatten_extralibs: 17s -> 0.2s > - FreeBSD/sh: total: 34s -> 9s flatten_extralibs: 23s -> 0.2s > - MSYS2/bash: total: 10m -> 2:30m flatten_extralibs: 400s -> 1.5s (!) > > > Notes: > > - unique() is modified with flatten_extralibs. It now outputs different > order: it was keeping the last instance of recurring items, now it > keeps the first. It affects libs order at ffbuild/config.{mak,sh} - > but I don't think it matters. If it does, "opt1-reorder-unique.patch" > restores the original order. Let me know if/why it matters and I'll > squash it and update the commit message accordingly if required. > > - After the check_deps() patch, pushvar() and popvar() are not used but > I was hesitant to remove them (nice to have). If you think it should > be removed, the patch "opt2-remove-pushvar.patch" removes them. > > - The patches assume POSIX shell and don't use anything "tricky". > It was tested with dash, bash, busybox-ash, freebsd-sh, ksh93u, mksh. > > - Thanks to tmm1 and atomnuker for their help with testing. > > > This is my first mail to this list, and I wasn't sure how to split the > patches. After asking a veteran, it was suggested to me that I should > put all of them in one email. Just let me know if you prefer something > else (dev mailing lists are not my forte...). with the main patch make distclean ; dash ./configure --enable-gpl && make -j12 testprogs fails: LD libavfilter/tests/filtfmts libavformat/libavformat.a(utils.o): In function `av_apply_bitstream_filters': ffmpeg/libavformat/utils.c:5577: undefined reference to `av_bitstream_filter_filter' libavformat/libavformat.a(codec2.o): In function `codec2_read_header_common': ffmpeg/libavformat/codec2.c:74: undefined reference to `avpriv_codec2_mode_bit_rate' ffmpeg/libavformat/codec2.c:75: undefined reference to `avpriv_codec2_mode_frame_size' ffmpeg/libavformat/codec2.c:76: undefined reference to `avpriv_codec2_mode_block_align' ffmpeg/libavformat/codec2.c:74: undefined reference to `avpriv_codec2_mode_bit_rate' ffmpeg/libavformat/codec2.c:75: undefined reference to `avpriv_codec2_mode_frame_size' ffmpeg/libavformat/codec2.c:76: undefined reference to `avpriv_codec2_mode_block_align' libavformat/libavformat.a(spdifdec.o): In function `spdif_get_offset_and_codec': ffmpeg/libavformat/spdifdec.c:63: undefined reference to `av_adts_header_parse' ffmpeg/libavformat/spdifdec.c:63: undefined reference to `av_adts_header_parse' libavformat/libavformat.a(spdifenc.o): In function `spdif_header_aac': ffmpeg/libavformat/spdifenc.c:357: undefined reference to `av_adts_header_parse' collect2: error: ld returned 1 exit status make: *** [libavfilter/tests/filtfmts] Error 1 [...]
Thanks. I'll post the 3 parts of main.patch unmodified as individualemails like Timo requested, and then I'll look at the failures. On Saturday, August 25, 2018 9:00 PM, Michael Niedermayer <michael@niedermayer.cc> wrote: On Sat, Aug 25, 2018 at 02:33:03PM +0000, avih wrote: > Hi, > > I noticed that configure can be a bit slow - few minutes on my macOS > system, 1:30m on linux/bash, 30s on linux/dash, and even if > pathological, still a too-long 10 minutes on Windows (MSYS2) (same > duration as a full build with make -j4 after configure). > > I added some timing information printouts for various parts in > configure, and eventually identified few culprits. > > The attachment "config-timing.patch" adds these printouts and sorted > summary. It's attached for reference and not intended to be merged. > It applies cleanly before or after the main patch. > > > The attached "main.patch" addresses three areas I identified as slow: > > 1. About 50-70% of configure runtime was being spent inside one > function: flatten_extralibs() and callees resolve() and unique(). > It manipulates strings and invoked nearly 20K (20000) subshells. > It was rewritten to avoid subshells, and ended up x50-x250 faster. > > 2. print_enabled_components() was invoking sed about 350 times on one > file. This is never instant but takes many seconds where fork is > slow (Windows). Invoke sed only once instead = x4-x10 speedup. > > 3. After the previous speedups, configure spent 20-60% of its runtime > at check_deps(). It's particularly slow with bash. After some local > optimizations - mainly avoid pushvar/popvar and abort early in one > notable case (empty deps), it's now x4-x25 faster. > > > Some resulting speedups (more details at "before-after.txt"): > > - macOS/bash: total: 98s -> 22s flatten_extralibs: 53s -> 0.7s > - Linux/bash: total: 87s -> 11s flatten_extralibs: 59s -> 0.9s > - Linux/dash: total: 27s -> 8s flatten_extralibs: 17s -> 0.2s > - FreeBSD/sh: total: 34s -> 9s flatten_extralibs: 23s -> 0.2s > - MSYS2/bash: total: 10m -> 2:30m flatten_extralibs: 400s -> 1.5s (!) > > > Notes: > > - unique() is modified with flatten_extralibs. It now outputs different > order: it was keeping the last instance of recurring items, now it > keeps the first. It affects libs order at ffbuild/config.{mak,sh} - > but I don't think it matters. If it does, "opt1-reorder-unique.patch" > restores the original order. Let me know if/why it matters and I'll > squash it and update the commit message accordingly if required. > > - After the check_deps() patch, pushvar() and popvar() are not used but > I was hesitant to remove them (nice to have). If you think it should > be removed, the patch "opt2-remove-pushvar.patch" removes them. > > - The patches assume POSIX shell and don't use anything "tricky". > It was tested with dash, bash, busybox-ash, freebsd-sh, ksh93u, mksh. > > - Thanks to tmm1 and atomnuker for their help with testing. > > > This is my first mail to this list, and I wasn't sure how to split the > patches. After asking a veteran, it was suggested to me that I should > put all of them in one email. Just let me know if you prefer something > else (dev mailing lists are not my forte...). with the main patch make distclean ; dash ./configure --enable-gpl && make -j12 testprogs fails: LD libavfilter/tests/filtfmts libavformat/libavformat.a(utils.o): In function `av_apply_bitstream_filters': ffmpeg/libavformat/utils.c:5577: undefined reference to `av_bitstream_filter_filter' libavformat/libavformat.a(codec2.o): In function `codec2_read_header_common': ffmpeg/libavformat/codec2.c:74: undefined reference to `avpriv_codec2_mode_bit_rate' ffmpeg/libavformat/codec2.c:75: undefined reference to `avpriv_codec2_mode_frame_size' ffmpeg/libavformat/codec2.c:76: undefined reference to `avpriv_codec2_mode_block_align' ffmpeg/libavformat/codec2.c:74: undefined reference to `avpriv_codec2_mode_bit_rate' ffmpeg/libavformat/codec2.c:75: undefined reference to `avpriv_codec2_mode_frame_size' ffmpeg/libavformat/codec2.c:76: undefined reference to `avpriv_codec2_mode_block_align' libavformat/libavformat.a(spdifdec.o): In function `spdif_get_offset_and_codec': ffmpeg/libavformat/spdifdec.c:63: undefined reference to `av_adts_header_parse' ffmpeg/libavformat/spdifdec.c:63: undefined reference to `av_adts_header_parse' libavformat/libavformat.a(spdifenc.o): In function `spdif_header_aac': ffmpeg/libavformat/spdifenc.c:357: undefined reference to `av_adts_header_parse' collect2: error: ld returned 1 exit status make: *** [libavfilter/tests/filtfmts] Error 1 [...]
On 08/25/18 11:11 AM, avih wrote: > After the previous speedups, configure spent 20-60% of its runtime > at check_deps(). It's particularly slow with bash. After some local > optimizations - mainly avoid pushvar/popvar and abort early in one > notable case (empty deps), it's now x4-x25 faster. Works great on OS/2, from 700 seconds to 144 seconds Dave
On 26-8-2018 22:27, Dave Yeo <daveryeo@telus.net> wrote: > On 08/25/18 11:11 AM, avih wrote: >> After the previous speedups, configure spent 20-60% of its runtime >> at check_deps(). It's particularly slow with bash. After some local >> optimizations - mainly avoid pushvar/popvar and abort early in one >> notable case (empty deps), it's now x4-x25 faster. > > Works great on OS/2, from 700 seconds to 144 seconds > Dave I've gone from 857s to 268s on my old WinXP pc. That's really impressive! -- Reino
From 77f897c8ed4eec9119d758037b0311629f549a5b Mon Sep 17 00:00:00 2001 From: "Avi Halachmi (:avih)" <avihpit@yahoo.com> Date: Wed, 1 Aug 2018 09:10:12 +0300 Subject: [PATCH] configure: remove unused pushvar()/popvar() --- configure | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/configure b/configure index e9cb7703..d6c1d032 100755 --- a/configure +++ b/configure @@ -619,25 +619,6 @@ get_sanitized(){ eval echo \$$(sanitize_var_name "$1") } -pushvar(){ - for pvar in $*; do - eval level=\${${pvar}_level:=0} - eval ${pvar}_${level}="\$$pvar" - eval ${pvar}_level=$(($level+1)) - done -} - -popvar(){ - for pvar in $*; do - eval level=\${${pvar}_level:-0} - test $level = 0 && continue - eval level=$(($level-1)) - eval $pvar="\${${pvar}_${level}}" - eval ${pvar}_level=$level - eval unset ${pvar}_${level} - done -} - request(){ for var in $*; do eval ${var}_requested=yes -- 2.17.1