13306 2019-04-24 21:20:57 +0000 bitbake starts up to n^2 processes with n cpus 2026-01-29 16:22:04 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.7 All Multiple NEW Medium+ normal 6.99 1 bunk randy.macleod liezhi.yang poky.bs.watcher poky.watcher randy.macleod richard.purdie tvgamblin --- --- Yes (doc changes required) oldest_to_newest 83634 0 bunk 2019-04-24 21:20:57 +0000 With n cores bitbake runs up to n tasks in parallel, and each might start up to n processes. E.g. on an 8-core CPU it is not uncommon that 8 compile tasks are running, each with 8 gcc processes. That's 64 processes competing for the same CPU/RAM/IO resources. There is no obvious simple solution that comes into my mind, but with n cpus the goal should be n cpu-bound processes and not up to n^2 processes. 83639 1 richard.purdie 2019-04-25 11:46:44 +0000 We did have a proof of concept for one idea to handle this: http://lists.openembedded.org/pipermail/openembedded-core/2015-February/101489.html Sadly we never finished debugging the issues it had. 83660 2 liezhi.yang 2019-04-26 02:24:04 +0000 (In reply to comment #0) > With n cores bitbake runs up to n tasks in parallel, and each might start up > to n processes. > > E.g. on an 8-core CPU it is not uncommon that 8 compile tasks are running, > each with 8 gcc processes. That's 64 processes competing for the same > CPU/RAM/IO resources. > > There is no obvious simple solution that comes into my mind, but with n cpus > the goal should be n cpu-bound processes and not up to n^2 processes. I've tried to adjust PARALLEL_MAKE and BB_NUMBER_THREADS several months ago, but didn't get a better result, the cpu count is nearly the best choice, you can try it if you like. The building process is a little complicated, not only related to CPU, but also memory and disk, and also parallel build (task dependencies). 83661 3 liezhi.yang 2019-04-26 02:27:43 +0000 (In reply to comment #1) > We did have a proof of concept for one idea to handle this: > http://lists.openembedded.org/pipermail/openembedded-core/2015-February/ > 101489.html > > Sadly we never finished debugging the issues it had. Is this similar to "make -l", please? -l [load], --load-average[=load] Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-point number). With no argument, removes a previous load limit. I've tried to add "-l <number>" to PARALLEL_MAKE several months ago, but didn't make much difference in build time, except found several parallel build issues, and I've fixed these issues in oe-core. 83662 4 liezhi.yang 2019-04-26 02:27:58 +0000 (In reply to comment #1) > We did have a proof of concept for one idea to handle this: > http://lists.openembedded.org/pipermail/openembedded-core/2015-February/ > 101489.html > > Sadly we never finished debugging the issues it had. Is this similar to "make -l", please? -l [load], --load-average[=load] Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-point number). With no argument, removes a previous load limit. I've tried to add "-l <number>" to PARALLEL_MAKE several months ago, but didn't make much difference in build time, except found several parallel build issues, and I've fixed these issues in oe-core. 89297 5 randy.macleod 2021-02-18 16:04:28 +0000 Robert, Is Richard's work mostly read to merge in your opinion? 89557 6 randy.macleod 2021-03-09 16:46:34 +0000 Future isn't the right target milestone. Set to 3.99 and CC RP. 96785 7 randy.macleod 2023-10-24 14:24:18 +0000 Bulk move from 4.99 or 0.00 to 5.99 104112 8 randy.macleod 2026-01-29 16:22:04 +0000 Randy and Martin Hundebøll have a patch. Randy is adding tests.