| Summary: | bitbake starts up to n^2 processes with n cpus | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Adrian Bunk <bunk> |
| Component: | bitbake | Assignee: | Randy MacLeod <randy.macleod> |
| Status: | NEW --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | liezhi.yang, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie, tvgamblin |
| Version: | 2.7 | ||
| Target Milestone: | 6.99 | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Yes (doc changes required) | |
|
Description
Adrian Bunk
2019-04-24 21:20:57 UTC
We did have a proof of concept for one idea to handle this: http://lists.openembedded.org/pipermail/openembedded-core/2015-February/101489.html Sadly we never finished debugging the issues it had. (In reply to comment #0) > With n cores bitbake runs up to n tasks in parallel, and each might start up > to n processes. > > E.g. on an 8-core CPU it is not uncommon that 8 compile tasks are running, > each with 8 gcc processes. That's 64 processes competing for the same > CPU/RAM/IO resources. > > There is no obvious simple solution that comes into my mind, but with n cpus > the goal should be n cpu-bound processes and not up to n^2 processes. I've tried to adjust PARALLEL_MAKE and BB_NUMBER_THREADS several months ago, but didn't get a better result, the cpu count is nearly the best choice, you can try it if you like. The building process is a little complicated, not only related to CPU, but also memory and disk, and also parallel build (task dependencies). (In reply to comment #1) > We did have a proof of concept for one idea to handle this: > http://lists.openembedded.org/pipermail/openembedded-core/2015-February/ > 101489.html > > Sadly we never finished debugging the issues it had. Is this similar to "make -l", please? -l [load], --load-average[=load] Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-point number). With no argument, removes a previous load limit. I've tried to add "-l <number>" to PARALLEL_MAKE several months ago, but didn't make much difference in build time, except found several parallel build issues, and I've fixed these issues in oe-core. (In reply to comment #1) > We did have a proof of concept for one idea to handle this: > http://lists.openembedded.org/pipermail/openembedded-core/2015-February/ > 101489.html > > Sadly we never finished debugging the issues it had. Is this similar to "make -l", please? -l [load], --load-average[=load] Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-point number). With no argument, removes a previous load limit. I've tried to add "-l <number>" to PARALLEL_MAKE several months ago, but didn't make much difference in build time, except found several parallel build issues, and I've fixed these issues in oe-core. Robert, Is Richard's work mostly read to merge in your opinion? Future isn't the right target milestone. Set to 3.99 and CC RP. Bulk move from 4.99 or 0.00 to 5.99 Randy and Martin Hundebøll have a patch. Randy is adding tests. |