Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
(CLOSED) Parallel make of kernel?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Helena
Veteran
Veteran


Joined: 02 Apr 2003
Posts: 1100
Location: Den Dolder, The Netherlands

PostPosted: Fri Sep 11, 2020 7:35 pm    Post subject: (CLOSED) Parallel make of kernel? Reply with quote

Just recently I found out that I could dramatically improve the time needed to build a new kernel by
Code:
make -j13
(I have an AMD Ryzen 5 2600 and use MAKEOPTS="-j13" too) instead of a simple "make". However, I do play around with my .config a lot. And now I find that rebuilding the kernel sometimes leads to (not so) funny behavior. In my case, USB support is either completely broken or breaks after starting X with
Code:
startx
In the latter situation the power button doesn't respond any more. So perhaps it's a DBUS problem, I don't know and frankly, I don't care, because when I redo everything using
Code:
make clean
first, I am back to normal again.

I never had these problems in the past before I started using parallel builds for the kernel. I have of course used MAKEOPTS>1 for years, so it can't be a compiler problem.

So my question is whether parallel building the kernel is bugfree... There have been been breaks in the past, apparently, e.g. 2 years ago.


Last edited by Helena on Sat Sep 12, 2020 7:26 pm; edited 1 time in total
Back to top
View user's profile Send private message
RayDude
Veteran
Veteran


Joined: 29 May 2004
Posts: 1725
Location: San Jose, CA

PostPosted: Fri Sep 11, 2020 10:33 pm    Post subject: Reply with quote

I've done make -j=n (where n>1) on the kernel for at least 10 years. I'm currently using -j16 for my Ryzen 7 4800H work laptop.

I've never had any problems with make with jobs set to large numbers on the kernel.

I suspect your problem is related to something else and somehow rebooting fixed the issue, not rebuilding the kernel.

Have you looked at dmesg and /var/log/messages? after starting X windows to see if there are any error messages?

You could also look at /var/log/Xorg.0.log
_________________
Some day there will only be free software.
Back to top
View user's profile Send private message
CaptainBlood
Veteran
Veteran


Joined: 24 Jan 2010
Posts: 1888

PostPosted: Fri Sep 11, 2020 11:22 pm    Post subject: Reply with quote

Here's
Code:
make clean
required at less than one a year rate, making me post here and getting straighten up by upstream advising to do so.
Thks 4 ur attention, interest & support.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2434
Location: Frankfurt, Germany

PostPosted: Fri Sep 11, 2020 11:43 pm    Post subject: Reply with quote

The number 13 is considered an unlucky number in some countries.

You may choose any number for 'make -j N'. But not 13! Sometimes that leads to broken kernels. :D
Back to top
View user's profile Send private message
Helena
Veteran
Veteran


Joined: 02 Apr 2003
Posts: 1100
Location: Den Dolder, The Netherlands

PostPosted: Sat Sep 12, 2020 1:24 pm    Post subject: Reply with quote

RayDude wrote:
I've done make -j=n (where n>1) on the kernel for at least 10 years. I'm currently using -j16 for my Ryzen 7 4800H work laptop.
Could it be that I need -j12 instead of -j13? Your approach suggest this. On the other hand, for emerging with MAKEOPTS=-j13 I have had no single problem so far. I know there is some debate about MAKEOPTS=”-j${core} +1" and the Gentoo wiki now recommends "The recommended value is the number of logical processors in the CPU. "
I'll give this a try first. :wink: @mike155
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 46364
Location: 56N 3W

PostPosted: Sat Sep 12, 2020 2:10 pm    Post subject: Reply with quote

Helena,

The 'optimum' value for X in MAKEOPTS="-jX" is no longer tied to just CPU cores.
Large values of X can lead to swapping because there is not enough real RAM to support X parallel make threads.

A few C++ packages need 2G RAM per thread, so gcc may want 26G RAM with -j13.
If you build in tmpfs too, you need a lot of RAM.

Swapping does not mean writing pages of dynamically allocated RAM to swap space. The kernel has other ways of swapping too.

I've built one or two kernels on a 96 core box using -j100. They seem to work.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Helena
Veteran
Veteran


Joined: 02 Apr 2003
Posts: 1100
Location: Den Dolder, The Netherlands

PostPosted: Sat Sep 12, 2020 5:59 pm    Post subject: Reply with quote

NeddySeagoon,

Thanks for clarifying this. I've noticed with my 16 GB RAM that make -j13 for the kernel never used more than 1.5 GB so that would be plenty. Moreover, swapping because of RAM shortage should not lead to errors, just take longer, right?

On the contrary, this would indicate that I could increase n in "make -j<n>" for kernel builds to the point where swapping starts to take place...
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 46364
Location: 56N 3W

PostPosted: Sat Sep 12, 2020 6:10 pm    Post subject: Reply with quote

Helena,

Thats right, unless the kernel invokes the Out of Memory Manager, which may kill your compile job. You notice that though.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Helena
Veteran
Veteran


Joined: 02 Apr 2003
Posts: 1100
Location: Den Dolder, The Netherlands

PostPosted: Sat Sep 12, 2020 6:13 pm    Post subject: Reply with quote

To further increase my curiosity, I should perhaps repeat and elaborate on what exactly happened the past few days/weeks with "make -j13":
  • When upgrading gentoo-sources (and therefore for the first time building this particular kernel version) I have no problems.
  • When I have changed my .config afterwards (especially, but perhaps not only, in the Drivers section) the build only recompiles kernel parts affected by my .config changes (presumably). This sometimes (more often than not) leads to a problematic reboot (I suspect the kernel is faulty but have not as yet definite proof)
  • When I precede the next kernel build by "make clean" the reboot with the new kernel is just fine.
So my theory is that the second step, an "incremental" kernel build, is not bugfree with "make -j<n>" when n>1. It may sound weird and I hope I am wrong.
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3809
Location: Hamburg

PostPosted: Sat Sep 12, 2020 6:32 pm    Post subject: Reply with quote

Helena wrote:
So my theory is that the second step, an "incremental" kernel build, is not bugfree with "make -j<n>" when n>1. It may sound weird and I hope I am wrong.
Whilst parallel build works fine here since a decade the kernel folks do recommend a "make clean" before reporting bugs. Given that maybe you actually run into a Makefile or a make depend issue. However -with "-j13" a "make clean" before should be ok for you.
Back to top
View user's profile Send private message
Helena
Veteran
Veteran


Joined: 02 Apr 2003
Posts: 1100
Location: Den Dolder, The Netherlands

PostPosted: Sat Sep 12, 2020 7:25 pm    Post subject: Reply with quote

toralf wrote:
Helena wrote:
So my theory is that the second step, an "incremental" kernel build, is not bugfree with "make -j<n>" when n>1. It may sound weird and I hope I am wrong.
Whilst parallel build works fine here since a decade the kernel folks do recommend a "make clean" before reporting bugs. Given that maybe you actually run into a Makefile or a make depend issue. However -with "-j13" a "make clean" before should be ok for you.
I welcome your reply and will do, as some others indeed have advised. However, that doesn't disprove my theory. As I cannot provide more proof yet I will mark this topic as closed.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum