Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TIP] Calculate your world compile time
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
lordoftheworld
n00b
n00b


Joined: 16 Jun 2004
Posts: 11

PostPosted: Tue Apr 10, 2018 3:47 am    Post subject: Reply with quote

Great info, I added a sort to see which packages might be taking up more time:

qlist -I | xargs qlop -t |sort -t" " -rnk2
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7006

PostPosted: Tue Apr 10, 2018 12:35 pm    Post subject: Reply with quote

Code:
 qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, $2 / 3600, ($2 % 3600) / 60, $2 % 60); }'
gzip:  1d:26h:52m:15s
perl:  0d:7h:48m:44s
libxslt:  0d:1h:41m:7s
gimp:  0d:6h:0m:47s
libpng:  0d:8h:18m:2s
wireshark:  0d:5h:19m:57s
diffutils:  4d:114h:42m:38s
findutils:  3d:81h:57m:18s
gawk:  0d:21h:10m:11s
grep:  0d:19h:58m:8s
kbd:  4d:110h:17m:48s
fuse:  14233598822306d:341606371735361h:55m:12s
e2fsprogs-libs:  0d:18h:23m:13s
perl-version:  0d:1h:50m:40s
lynx:  1d:25h:54m:59s

yep something is weird, thanks for the update
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 227
Location: Brazil

PostPosted: Tue Apr 10, 2018 5:36 pm    Post subject: Reply with quote

krinn wrote:
Code:
 qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, $2 / 3600, ($2 % 3600) / 60, $2 % 60); }'
gzip:  1d:26h:52m:15s
perl:  0d:7h:48m:44s
libxslt:  0d:1h:41m:7s
gimp:  0d:6h:0m:47s
libpng:  0d:8h:18m:2s
wireshark:  0d:5h:19m:57s
diffutils:  4d:114h:42m:38s
findutils:  3d:81h:57m:18s
gawk:  0d:21h:10m:11s
grep:  0d:19h:58m:8s
kbd:  4d:110h:17m:48s
fuse:  14233598822306d:341606371735361h:55m:12s
e2fsprogs-libs:  0d:18h:23m:13s
perl-version:  0d:1h:50m:40s
lynx:  1d:25h:54m:59s

yep something is weird, thanks for the update
Thanks for pointing out my mistake krinn, there was really an error. Since the format of the output from this command is d:h:m:s hours should be always less than 24 (remainder from 86400 seconds division), like minutes (remainder from 3600 seconds division) and seconds (remainder from 60 seconds division) are less than 60. $2 / 3600 should be ($2 % 86400) / 3600

I updated again the first post.
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 227
Location: Brazil

PostPosted: Tue Apr 10, 2018 6:58 pm    Post subject: Reply with quote

<3 wrote:
can we make an ebuild of this to put in portage or add this to gentoolkit?
I think it would be suitable for qlop from q applets.

I can open a bug report asking for this feature if people want to.
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
josephg
l33t
l33t


Joined: 10 Jan 2016
Posts: 782
Location: usually offline

PostPosted: Fri Apr 13, 2018 10:18 pm    Post subject: Re: [TIP] Calculate your world compile time Reply with quote

rudregues wrote:
EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.

1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation


Why would you want to remove bigger packages? Those are the ones which matter for this purpose.

Something still doesn't look right :?:
Code:
$ qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'
sudo:  1d:18h:35m:44s
libreoffice:  0d:12h:38m:34s
libnl:  0d:1h:56m:27s
libpcre:  0d:1h:46m:3s
qtdbus:  0d:1h:51m:35s
qtprintsupport:  0d:2h:42m:36s
qtwebengine:  0d:19h:4m:1s
qtwebkit:  6d:18h:9m:2s
qtxmlpatterns:  0d:2h:0m:40s
terminus-font:  1d:13h:15m:10s
opus:  0d:2h:22m:54s
ffmpeg:  0d:10h:3m:37s
libpcap:  6d:2h:7m:37s
busybox:  0d:2h:4m:8s
man-db:  0d:8h:59m:9s
man-pages:  0d:1h:47m:35s
gcc:  0d:3h:46m:6s
llvm:  0d:2h:16m:34s
gentoo-sources:  0d:16h:49m:9s
kergen:  4d:3h:17m:40s
e2fsprogs-libs:  0d:2h:11m:54s
package-manager:  1d:22h:54m:13s
service-manager:  1d:22h:31m:27s
ssh:  1d:22h:50m:52s
firefox:  0d:8h:38m:9s
adwaita-icon-theme:  0d:3h:51m:9s

_________________
"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey
Back to top
View user's profile Send private message
<3
l33t
l33t


Joined: 21 Oct 2004
Posts: 918

PostPosted: Sat Apr 21, 2018 8:05 am    Post subject: Reply with quote

rudregues wrote:
<3 wrote:
can we make an ebuild of this to put in portage or add this to gentoolkit?
I think it would be suitable for qlop from q applets.

I can open a bug report asking for this feature if people want to.


that would be great
Back to top
View user's profile Send private message
edgar_uriel84
Guru
Guru


Joined: 21 Dec 2008
Posts: 473
Location: Tultitlan, México

PostPosted: Tue May 01, 2018 7:40 pm    Post subject: Reply with quote

It's funny

Code:
┌─ [4966][14:24:27][genomorro@genomorro-xps-15-9550][~]
└ᐅ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
7h:31m:19s
┌─ [4966][14:29:10][genomorro@genomorro-xps-15-9550][~]
└ᐅ qlist -I | xargs qlop -t | awk '{ if ($2 > 1800) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'
libreoffice:  0d:1h:4m:54s
┌─ [4966][14:29:19][genomorro@genomorro-xps-15-9550][~]
└ᐅ uname -a
Linux genomorro-xps-15-9550 4.9.76-gentoo-r1 #1 SMP PREEMPT Mon Apr 2 01:56:45 CDT 2018 x86_64 Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz GenuineIntel GNU/Linux


I'm using DWM as WM.
_________________
Algún día habrá una persona mejor que yo, pero esa persona no eres tú ni ese momento es ahora.
Back to top
View user's profile Send private message
Nreal
Apprentice
Apprentice


Joined: 06 Jan 2009
Posts: 150

PostPosted: Sat Jun 16, 2018 12:43 pm    Post subject: Reply with quote

Two of those bastards Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz is together 72 threads, (36 actual cores)

Code:
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
1024819115206107h:1m:20s


Waiting for more ram as 16gt swaps with anything above -j30.. Its fast though :)

Code:
boxi /home/markus # uname -a
Linux boxi 4.17.1-gentoo #6 SMP PREEMPT Sat Jun 16 12:55:42 EEST 2018 x86_64 Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz GenuineIntel GNU/Linux

processor   : 71
vendor_id   : GenuineIntel
cpu family   : 6
model      : 63
model name   : Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz
stepping   : 2
microcode   : 0x3c
cpu MHz      : 2793.850
cache size   : 46080 KB
physical id   : 1
siblings   : 36
core id      : 27
cpu cores   : 18
apicid      : 119
initial apicid   : 119
fpu      : yes
fpu_exception   : yes
cpuid level   : 15
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts
bugs      : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips   : 4590.71
clflush size   : 64
cache_alignment   : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


[Moderator edit: added [code] tags to preserve output layout. -Hu]
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 227
Location: Brazil

PostPosted: Sat Jun 23, 2018 1:57 pm    Post subject: Reply with quote

I posted a bug report asking for add this functionality into qlop (app-portage/portage-utils) https://bugs.gentoo.org/658824
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
phelix
n00b
n00b


Joined: 28 Oct 2018
Posts: 2

PostPosted: Sun Oct 28, 2018 2:16 am    Post subject: Reply with quote

Hi there,

have you tried genlop?

Code:

$ emerge --pretend --emptytree @world | genlop --pretend
[...]
Estimated update time: 5 hours, 39 minutes.


cheers

-- edited for formatting
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 227
Location: Brazil

PostPosted: Fri Nov 16, 2018 1:21 pm    Post subject: Reply with quote

phelix wrote:
Hi there,

have you tried genlop?

Code:

$ emerge --pretend --emptytree @world | genlop --pretend
[...]
Estimated update time: 5 hours, 39 minutes.


cheers

-- edited for formatting
Your tip worked, but it has some caveats:
1) It took a big while to complete (maybe 1 minute?);
2) The output is very verbose because of emerge (using --quiet hasn't effect to minimize this);
3) Emerge will look into new packages to satisfy dependencies, not only the already installed, maybe increasing the time in the answer.

Other than these problems, the final answer was Estimated update time: 1 day, 2 hours, 4 minutes., which is pretty close to my script answer of 25h:1m:32s
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
ppurka
Advocate
Advocate


Joined: 26 Dec 2004
Posts: 3256

PostPosted: Wed Jan 02, 2019 2:36 pm    Post subject: Reply with quote

qlop seems unreliable for some cases. My script (see my signature) provides an estimated time
Code:
quietemerge -pe1 @world
...
Total ETA: 1d 16h 24m 10s
The command in the main post provides an estimated time of over 10 days
Code:
~» qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
290h:22m:38s


It seems that some of the estimates made by qlop are incorrect:
Code:
 » qlop -v -g -H qtcurve
qtcurve-1.8.17-r1: Sat Mar 22 11:19:38 2014: 1 minute, 19 seconds
qtcurve-1.8.17-r1: Thu Apr 24 04:27:08 2014: 39 seconds
qtcurve-1.8.17-r1: Sun Oct 19 22:44:02 2014: 51 seconds
qtcurve-1.8.18-r1: Sat Feb 20 19:07:04 2016: 2 minutes, 16 seconds
qtcurve-1.8.18-r1: Sun May  8 14:10:10 2016: 2 minutes, 8 seconds
qtcurve-1.8.18-r1: Sun May  7 22:41:07 2017: 104 days, 13 hours, 34 minutes, 53 seconds
qtcurve-1.9.0_rc1-r1: Sun Oct  1 16:37:44 2017: 2 minutes, 20 seconds
qtcurve-1.9.0_rc1-r1: Sat Nov 11 17:14:05 2017: 1 minute, 32 seconds
qtcurve-1.9.0_rc1-r1: Mon Jul  2 19:38:20 2018: 1 minute, 7 seconds
qtcurve-1.9.0_rc1-r2: Sun Jul 15 19:13:07 2018: 1 minute, 50 seconds
qtcurve-1.9.0: Sun Oct 21 20:23:40 2018: 2 minutes, 20 seconds
qtcurve: 11 times
On the other hand, the estimates provided by the internal computations in the quietemerge script gives the following (in reverse order to above):
Code:
~ » mygenlop qtcurve
 *  Printing the times in reverse order (recent emerge appears first)
    Run no.  Time Taken
         1    2m 20s
         2    1m 50s
         3    1m 7s
         4    1m 32s
         5    2m 20s
         6    1m 2s
         7    2m 8s
         8    2m 16s
         9    51s
        10    39s
        11    1m 19s
 *  Average time taken over 11 runs: 1m 34s

_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/
Back to top
View user's profile Send private message
pcmaster
Advocate
Advocate


Joined: 04 Dec 2003
Posts: 2153
Location: Barcelona

PostPosted: Wed Jan 30, 2019 7:06 pm    Post subject: Reply with quote

In my box:

Code:
Core2Duo ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
42h:19m:8s
Core2Duo ~ #


Intel Core2 Duo 6550
4 GB RAM
2 TB WD black HDD
_________________
pcmaster
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1477
Location: KUUSANKOSKI, Finland

PostPosted: Wed Jan 30, 2019 11:07 pm    Post subject: Reply with quote

I don't know if some else has already mentioned this, but the estimation the command gives is almost always larger than the actual world compile time, unless one disallows parallel merging of packages (emerge --jobs <n>).
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
proxy
Apprentice
Apprentice


Joined: 20 Apr 2002
Posts: 260
Location: Chantilly, VA

PostPosted: Thu Feb 14, 2019 3:10 pm    Post subject: Reply with quote

Zucca wrote:
I don't know if some else has already mentioned this, but the estimation the command gives is almost always larger than the actual world compile time, unless one disallows parallel merging of packages (emerge --jobs <n>).


I was thinking the same thing. Though, you may be able to get a sensible estimate by dividing the result by the number of cores you have (assuming they are all the same spec). Won't be precise, but would probably be close enough to be informative as a real world estimate.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1477
Location: KUUSANKOSKI, Finland

PostPosted: Fri Feb 15, 2019 1:36 pm    Post subject: Reply with quote

Any bigger package that can run make -j<N> effectively at compile phase would normally limit emerge --jobs mto just one.
For example I have --jobs 8 --load-average 10.00 on my EMERGE_DEFAULT_OPTS.
So maximum of 8 concurrent package installs/compiles, but only is load average is under 10.00.

So sometimes for example Firefox and libreoffice both get to be compiled at the same time, effectively lenghtening the compile time of both on my 8-core CPU (would result around load average of 16). In this case compile times recorded for these packages are 2x at worst.

On the other hand some small packages that really can't fully utilize make -j8 (on my system), are almost always being run at 1x speed, but those can be "emerged" parallel (at most of 8 packages being compiled/installed at the same time).
In this case it's the opposite to the previous one. Since on average all those smaller packages are "emerged" in "time * 1/8". But the method counting the world compile time gives each 1x time instead of 1/8.

So you can't just divide the time by the cores on your system... unfortunately.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum