Bug 6974 - bitbake hangs suddenly, child processes frozen in stopped state
Summary: bitbake hangs suddenly, child processes frozen in stopped state
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 1.6.2
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 1.8 M2
Assignee: Richard Purdie
QA Contact: Costin
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2014-11-17 21:19 UTC by edi
Modified: 2015-01-13 10:31 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Sample of the process tree (5.62 KB, text/plain)
2014-11-17 21:19 UTC, edi
no flags Details
Console log (3.25 KB, text/plain)
2014-11-17 21:30 UTC, edi
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description edi 2014-11-17 21:19:03 UTC
Created attachment 2249 [details]
Sample of the process tree

I am trying to build Yocto 1.7 or 1.6 for an arm a20 board. Running on Debian Unstable amd64 system, quad-core AMD cpu, 12gb RAM, ext4 everywhere, nothing special here.

In the middle of the process, it get stuck while building the linux recipe from the meta-sunxi layer. I had a look at the recipe and cannot see anything causing a such hang. Looking at the process tree shows something weird, a lot of processes simply stuck in T state for no obvious reason. I can send SIGCONT to all of them with a script and after a second the process tree gets frozen again.

Just have a look at the attached sample, it looks totally unnatural. Only SIGKILLing python processes ends this mess, SIGTERM is not effective.

Just guessing, based on previous experience with OE: is this maybe some kind of black magic in bitbake, maybe trying to balance the execution to avoid out-of-memory situations and making it worse instead?
Comment 1 edi 2014-11-17 21:30:02 UTC
Created attachment 2250 [details]
Console log
Comment 2 edi 2014-11-20 13:31:14 UTC
PS: I added toaster to the config as documented in https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster#Bitbake_extra_options and after that the problem did not appear. Still cannot explain how it happens.
Comment 3 Richard Purdie 2014-12-06 09:42:09 UTC
There is no black magic in bitbake that handles this kind of situation. Some kind of signals must be being passed to the subprocesses but this isn't normal behaviour and isn't something I can replicate. Was this similar to https://bugzilla.yoctoproject.org/show_bug.cgi?id=6949 where Ctrl+C had been pressed as an initial trigger?
Comment 4 edi 2014-12-06 14:15:52 UTC
Well, this problem might be the actuall trigger for #6949, at least if provides a good way to reproduce it (a forced building the sunxi kernel). Most likely it was there all the time in background and I didn't notice the first time when I opened the other bug report. I suggest to merge this reports, if possible.

In the meantime I made a couple of experiments, "downgraded" the system to official Jessie kernel and sysvinit. And it was still reproducible so I asssume that kernel features or pid1 don't really have much to do with it. This report was made using Terminator, and I also used plain uxterm a couple of times, IMHO it might be caused by terminal settings but I have doubts.

The next experiment would be building a pristine VM and test it there, and then install the package set of my workstation and try with them, etc... there must be something that triggers this behaviour.

Let's see when I find time to do it.
Comment 5 Richard Purdie 2015-01-08 09:48:59 UTC
I've merged http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=130958cc24ad306684d92a8d965f3772f30e196a since it appears to be the issue, even if we don't understand the trigger. At this point this is the best I can do. If we do have any more information please do reopen the bug or comment as appropriate.