From patchwork Sat Aug 25 18:42:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: avih X-Patchwork-Id: 10137 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:12c4:0:0:0:0:0 with SMTP id 65-v6csp4736805jap; Sat, 25 Aug 2018 11:43:01 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbfRYgUIJzoR7KUd5sezKzLGgutY7hQA+jL3DU3x1RL4Qu8FPkVCjhk2QmckfjA5HKX3irL X-Received: by 2002:a1c:4e16:: with SMTP id g22-v6mr1783342wmh.111.1535222581416; Sat, 25 Aug 2018 11:43:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535222581; cv=none; d=google.com; s=arc-20160816; b=cZJhIzTfZEMN7hECY/hDA7B62XZi5TK3hTNO08ev2KzELepndEPgJ4A1uOyPAJsPi2 fdfJa1EgNU9OL3S/C7jwRRB4cYyjnWbkgnYL+KwGigVXfN787brlM1OSYlgCwFAooW4Z 49F7+8Q1axsMOVW1QjdGVovUuzIq1fvvk0uiQZeRT2UprI0gdihwr8BJhH37/Gw9jLLb QcOs0dTx2hSORfsRcLcvi8l96qD+Mbuw29kb+PmrghjT8xImBv1LUBlrPG/pOQtJatqo 36cjGh9oQ0ufSPCPx62ADbBWys8wTxi4AATJbQSgbwDMQ42f+dfE+LVC3UIAh1nqz/1X mrRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:to:from:date :dkim-signature:delivered-to:arc-authentication-results; bh=3Mbhyr7CeTytI3cHxzNXN021sO0I5xVY7hQusWZeNtw=; b=ymzET04D3gmAvgDbSFsswy7aalWAVxDHlSdDVPdqSoKBYhd66IlB3hFwsobvwcBZkh BVVv6Galr2tFIWatKR2ibQNW8L2YgtHyJi0KTeBNdyLmtkRb/IH4DV5Ne2Jwkjwa/9L9 f2p+StRaGi/BCm0l2y8f3L+JMh0EVw+pvKsZswREcPnuv5jkHeeEmJ8jZJFBqqbySNPb ld1KBn23DCTPugwdQooboIZnzy1f93pZQAMZu2ZZIwnu9HdZJD009ZZ4ITWFeEvpZUWC fQX4TXVrbd9R9tPGA8UL7DM1IS0TpPCLmg/8It9h5PYQaw9I/HFIF+b78fUJQ7HPDiQE gz4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@yahoo.com header.s=s2048 header.b=MIQcqM9s; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m12-v6si9520743wrg.369.2018.08.25.11.43.01; Sat, 25 Aug 2018 11:43:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@yahoo.com header.s=s2048 header.b=MIQcqM9s; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4E1DD689AF6; Sat, 25 Aug 2018 21:42:57 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from sonic312-21.consmr.mail.bf2.yahoo.com (sonic312-21.consmr.mail.bf2.yahoo.com [74.6.128.83]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C1F9A6806C2 for ; Sat, 25 Aug 2018 21:42:50 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1535222571; bh=8Ria+NeQc8nn3FFJ2A+ixCYKqiu0DCLanOKatRTEzZQ=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=MIQcqM9sCKu7xb3Z/tBgw7BKgCDAwPOVFGKlBo0oBNOXHyZPuayYg22OniXQzh+JwVHwZOee7WH6O5okaLsh8mNzCAFVDPDUboC4HifEqXBy656NHrW2ntjhj2ul/z+o4RMO6b1Iovm1BU/8bvaMLPMAKlBSIgmVK9i8eQ5OSug2XMZkFbEHJSDGFzczSrcAmLZvKWnEh9BSXkwkw2z/o0hVfH/A/khbJSky2MPtQ3OKvTV9ZtJS013L2fz9YKjUnXKa7UiGojd0+dotBxBfaIZfJ0+qoXBXGybeni4QLm16V4ezMvwhEiWOIWtENiz148pSfSiMHC9UiV1o2K+BwQ== X-YMail-OSG: Q7PDnHIVM1keXxklqGUWpM_ioA7zhdlpBSm5018PdgSzbPnnrMOZb_xuVSVf6ib B_9rzvjVOZyxwaiSooBvAOYUlCDtcEcjJyN18f6RbFHxO1Fmg25tUyFcDoyquwQgo1KHJxc.pO8q 59B.SfhZ_9.ZUxVwKAHvVMuMyPPIYAQvBN6whRPZkloRr.LedBwJ0scp5kZuN78fpuMMm8F6sFq8 N35tU36bnWNtOsDzJl36c.dysu0MDMBWKXGJNcRXX7XkD5He_6kmYEsm2kRJCFG1W2ofaj1v8JPl BJdDdRhkUSceYts0c5H6MkenFootN9yAFFRp51LW4pkamodpD9a7pEk8QnsGW5ptR0mWHUp4FTH9 tpR1KYLWWzFKlCcsz2l9Omctx.oZR8VGlBjFWbzmeO_aISkeNb0nbcWMINaA9chQ8qsEREL_8VKq qowqOI_kFuk.o50.RfnImo5JbH1SZAV0c6IjTOjEWwgeHgYm.Xj3Lb8.OnHVekU.vnkq6LufK6At UTv_YO3mua4Tw_P.ilDic.opAEhU5AMBSlDZD7JiYc7KQqtO2Ab7toRpMRnHnEPtTVbEImAWTWVK fwfDxmskBJNKPjKi7kp3u7UopbpuC5HDcAN91LwHXPFK5i0HB2IIdHgOOq5qcbyX0wJmWA5MOBAK Pcbem1KkjyQbToaq7WrDJNGe8vw4BIZkZiUYVTZGXV4KTdoRlstg5pOhzG0AzqHzbW6nLjqdznTF tnN798tSL2d9ocAc9XG6yCwFFxzcXq18oiI8xoxihSdT13KDJbTvXKFB5b.CSLq243Xo2umu7I_U WxzwluTcts8OWGJkPg6E.pKjUm_SvonPBvnLPFWIptduyRP0gwxDmBG9xG2l6OKsBEQ4._X3_kKV aYzPJIP14mukwLCb_SlwqQB30HfVrFGR4cjYF9Kuc4_mdkYtLJcsdYpEJDRX8uDSWDEiA.xWpzyB nxtxrV7yEd25qDGhJmA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.bf2.yahoo.com with HTTP; Sat, 25 Aug 2018 18:42:51 +0000 Date: Sat, 25 Aug 2018 18:42:48 +0000 (UTC) From: avih To: FFmpeg development discussions and patches Message-ID: <1829695586.2576194.1535222568926@mail.yahoo.com> In-Reply-To: <996141963.2585533.1535220535147@mail.yahoo.com> References: <378315057.2531045.1535207584068.ref@mail.yahoo.com> <378315057.2531045.1535207584068@mail.yahoo.com> <996141963.2585533.1535220535147@mail.yahoo.com> MIME-Version: 1.0 X-Mailer: WebService/1.1.12262 YahooMailNeo Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 Subject: Re: [FFmpeg-devel] [PATCH 1/3 v2] configure: speedup x2-x8 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" It turns out the the output order of "unique()" was important, andfailing to keep the original order (where it keeps the last instanceof recurring items) caused some linkage failures. This Patch 1/3 v2 squashes opt1-reorder-unique.patch onto the originalpatch 1/3, and updates the commit message accordingly. It fixes the linkage failures mentioned at:http://ffmpeg.org/pipermail/ffmpeg-devel/2018-August/233674.html On Saturday, August 25, 2018 9:09 PM, avih wrote: For overview see: http://ffmpeg.org/pipermail/ffmpeg-devel/2018-August/233665.html Attached is patch 1/3 which was at "main.patch" of that message. About 50-70% of configure runtime was being spent inside one function: flatten_extralibs() and callees resolve() and unique(). It manipulates strings and invoked nearly 20K (20000) subshells. It was rewritten to avoid subshells, and ended up x50-x250 faster. unique() now outputs a different order: it was keeping the last instance of recurring items, now it keeps the first. It affects libs order at ffbuild/config.{mak,sh} - but I don't think it matters. If it does, "opt1-reorder-unique.patch" restores the original order. Let me know if/why it matters and I'll squash it and update the commit message accordingly if required.     On Saturday, August 25, 2018 7:55 PM, Timo Rothenpieler wrote: Please use git send-email to send your patches, or at least send each patch, created by git format-patch, as individual attachment. Your files seem to contain multiple patches one after another, which makes them very hard to follow. But nice work! Let's hope this does not cause any regressions. From 17b42195afb8e3cdd1fd03c5b14634c64c78a1bc Mon Sep 17 00:00:00 2001 From: "Avi Halachmi (:avih)" Date: Mon, 30 Jul 2018 22:39:15 +0300 Subject: [PATCH] configure: flatten_extralibs_wrapper(): speedup x50 - x200 Currently configure spends 50-70% of its runtime inside a single function: flatten_extralibs[_wrapper] - which does string processing. During its run, nearly 20K command substitutions (subshells) are used, including its callees unique() and resolve(), which is the reason for its lengthy run. This commit avoids all subshells during its execution, speeding it up by about two orders of magnitude, and reducing the overall configure runtime by 50-70% . resolve() is rewritten to avoid subshells, and in unique() and flatten_extralibs() we "inline" the filter[_out] functionality. Note that logically, "unique" functionality has more than one possible output (depending on which of the recurring items is kept). As it turns out, other parts expect the last recurring item to be kept (which was the original behavior of uniqie()). This patch preservs its output order. --- configure | 46 +++++++++++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 13 deletions(-) diff --git a/configure b/configure index b9c9d0b3..95a38cf2 100755 --- a/configure +++ b/configure @@ -844,22 +844,37 @@ prepend(){ eval "$var=\"$* \$$var\"" } +reverse () { + eval ' + reverse_out= + for v in $'$1'; do + reverse_out="$v $reverse_out" + done + '$1'=$reverse_out + ' +} + +# keeps the last occurence of each non-unique item unique(){ - var=$1 - uniq_list="" - for tok in $(eval echo \$$var); do - uniq_list="$(filter_out $tok $uniq_list) $tok" + unique_out= + eval unique_in=\$$1 + reverse unique_in + for v in $unique_in; do + # " $unique_out" +space such that every item is surrounded with spaces + case " $unique_out" in *" $v "*) continue; esac # already in list + unique_out="$unique_out$v " done - eval "$var=\"${uniq_list}\"" + reverse unique_out + eval $1=\$unique_out } resolve(){ - var=$1 - tmpvar= - for entry in $(eval echo \$$var); do - tmpvar="$tmpvar $(eval echo \$${entry})" + resolve_out= + eval resolve_in=\$$1 + for v in $resolve_in; do + eval 'resolve_out="$resolve_out$'$v' "' done - eval "$var=\"${tmpvar}\"" + eval $1=\$resolve_out } add_cppflags(){ @@ -6734,14 +6749,19 @@ if test $target_os = "haiku"; then fi flatten_extralibs(){ - unset nested_entries + nested_entries= list_name=$1 eval list=\$${1} for entry in $list; do entry_copy=$entry resolve entry_copy - append nested_entries $(filter '*_extralibs' $entry_copy) - flat_entries=$(filter_out '*_extralibs' $entry_copy) + flat_entries= + for e in $entry_copy; do + case $e in + *_extralibs) nested_entries="$nested_entries$e ";; + *) flat_entries="$flat_entries$e ";; + esac + done eval $entry="\$flat_entries" done append $list_name "$nested_entries" -- 2.17.1