Bug 3246

Summary: Set more useful defaults for BB_NUMBER_THREADS and PARALLEL_MAKE
Product: [Build System, Metadata & Runtime] BitBake Reporter: Evadeflow <evadeflow>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED DUPLICATE QA Contact:
Severity: enhancement    
Priority: Undecided CC: poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: ---   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Evadeflow 2012-10-06 15:12:49 UTC 
While executing some git commands this morning, I noticed that it
decided all by itself how many threads it could use for delta
compression:

> $ git push --mirror github
> Counting objects: 162905, done.
> Delta compression using up to 4 threads.
> Compressing objects: 100% (42808/42808), done.
> Writing objects:  25% (41080/162905), 9.31 MiB | 102 KiB/s

It would be nice if bitbake could do the same, i.e., introspect the
system to determine how many cores are available and try to make the
best use of them. Then users could put this in local.conf:

> BB_NUMBER_THREADS = "${NUM_CORES}"
> PARALLEL_MAKE = "${NUM_CORES}"

Also, I think bitbake defaults to "1" for both of these if not set?
Since that gives an excruciatingly slow user experience, maybe some
heuristic could be used to pick a more sensible default value, something
along the lines of:

  1 if ${NUM_CORES} = 1
  2 if ${NUM_CORES} = 2
  2 if ${NUM_CORES} = 3
  ${NUM_CORES}/2 otherwise

(The idea is not to bring the machine to its knees if the user doesn't
explicitly specify "${NUM_CORES}" in local.conf.)

Just thinking out loud. All I know is: the current defaults for these
variables are totally unsuitable for real-world use, and it's tricky for
a new user to figure out what the 'right' values should be.
Comment 1 Evadeflow 2012-10-08 02:56:46 UTC 
Looking into this a bit more, I see that this info is readily available
from python:

>>> import multiprocessing
>>> multiprocessing.cpu_count()
4

So maybe adding Yet Another Variable isn't necessary. Instead, just set
values for BB_NUMBER_THREADS and PARALLEL_MAKE using a python snippet
that can be overridden by the user?
Comment 2 Richard Purdie 2012-10-08 08:15:25 UTC 
I don't think a NUM_CORES variable makes sense since that data is already available to the system as you later mention. I'm also reluctant to start doing magic things when the user making the build should probably make a concious choice about what settings to use. There is already an ehancement request about automatic settings of these variables so making this as a duplicate of that.

*** This bug has been marked as a duplicate of bug 2528 ***