Bug 11802 - system spontaneously logs out while building core-image
Summary: system spontaneously logs out while building core-image
Status: RESOLVED WORKSFORME
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 4.99
Assignee: Mingli Yu
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-07-17 20:41 UTC by Armin Kuster
Modified: 2022-02-17 16:11 UTC (History)
11 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments
syslog (2.22 KB, text/plain)
2017-07-20 10:33 UTC, Rebecca Chang
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Kuster 2017-07-17 20:41:34 UTC
This is more of a place to capture information to see if we can determine a cause of this issue.

I see this issue more on laptops while building. I has happened as far back as krogoth. What happens is during a build, the desktop will log me off. No messages to be found anywhere. I has happened on two different laptops on two different Ubuntu distros.

This was discussed on the ML a few months back with no solution as far as I can tell.

Not sure how prevalent this is. I don;t know if it worth collecting info to try to understand this issue?
Comment 1 Juro Bystricky 2017-07-19 20:28:40 UTC
What happens to me on a regular basis (on a fairly powerful desktop) is that my X server crashes, then forces me to login again. In my /var/log/syslog I see something like: 

XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"

followed by bunch of apps crashing due to X11 running of resources.
I can make this happen quite easily just by running two parallel core-images, typically from scratch (no SSTATE). Very annoying.
I experienced this with Ubuntu 15 & 16, two different desktops (8 threads and 40 threads).
Comment 2 Saul Wold 2017-07-20 00:04:17 UTC
We first need to create a reproducible case, possibly by forcing, or using somekind of container spawning system to reproduce this.
Comment 3 Rebecca Chang 2017-07-20 10:33:53 UTC
Created attachment 3885 [details]
syslog
Comment 4 Rebecca Chang 2017-07-20 10:45:55 UTC
This happen on my workstation randomly from time to time. But I was able to reproduce it 3 times in a row while building SDK.

$ bitbake core-image-minimal -c populate_sdk

Not sure what is the root cause yet, But I do get the same errors on X server failed. I also wanted to share that, my workstation was a shared machine for a few developers. The issue should be reproduce on an environment that do not has any other interference. However Stanley did tried the same command on his personal laptop but unable to reproduce with single build running.

My workstation: Ubuntu 16, 40 threads 32GB RAM
Comment 5 Paul Eggleton 2017-07-25 07:21:31 UTC
Is this the OOM killer at work in some of these cases? Anything indicative in the system logs?
Comment 6 Rebecca Chang 2017-07-25 08:05:28 UTC
system log mentioned about X server failure causing chain effect to kill and restart the gnome session.
I have recently changed my window manager to budgie desktop and the symptom no longer be reproducible. I still not sure what is the actual root cause.
Comment 7 Juro Bystricky 2017-07-26 02:45:13 UTC
I have always (perhaps unfairly) suspected pseudo leaking resources. But I could not find a smoking gun. Now I use text console (CTRL-ALT-F1), no problems there. Also no problems in Ubuntu container, but that also runs without GUI/X11. OOM is not likely, as I have tons of unused memory.
Comment 8 Juro Bystricky 2017-07-31 22:16:56 UTC
I should mention that replacing video cards (Nvidia/Radeon,...) and display drivers (private, open source) did not make any difference. (This being often cited as remedy when Googling).
Comment 9 Juro Bystricky 2017-08-08 23:34:50 UTC
So today this happened again. One thing worth mentioning is that I killed a runnning "bitbake core-image-minimal"  with CTRL-C running in "konsole". 
Moments later I tried to re-run bitbake core-image-minimal and X server crashed practically right away.
Comment 10 Mingli Yu 2018-05-25 07:20:34 UTC
I didn't encounter this, could you help to provide the detailed reproduce steps please?
Comment 11 Randy MacLeod 2018-05-25 13:45:41 UTC
@mingli, see the previous descriptions. The challenge with this bug is that no one is sure exactly how to reproduce it. 

For anyone who reported a failure, can you describe the system (cores, ram, storage system, free -m, ulimit -a, X session or server)? I suspect but don't really have any proof that resource limits (file descriptors) are the root cause and as people suggested, the X server fails after the resources are exhausted.

I couldn't find the discussion thread that Armin mentioned so if someone could post a link to that, it might help.
Comment 12 Juro Bystricky 2018-05-25 14:03:38 UTC
It is very easy to reproduce on my 44 or 88 core Ubuntu 16.04 systems.
All I need to do is to attempt a clean build of core-image-sato. This will eventually result in the problem as described. The problem is so common, I only build images in terminal sessions (Ctrl-Alt-Fn).
Comment 13 Armin Kuster 2018-09-28 00:49:42 UTC
when I saw this was via my 2 core laptop. I have not seen is a while. maybe since I stopped build morty for OE/Yocto
Comment 14 Ross Burton 2018-10-01 19:54:44 UTC
To anyone who can still replicate this: my gut feeling is the same as Randy: some resource is being limited and the X server either dies, or loses the OOM roulette and gets killed.  Watching the syslog and various resource limits during a build would be the way to debug this.
Comment 15 ran 2018-12-19 22:11:15 UTC
I have the same issue:

laptop: lenovo t460p
using virtual box with ubuntu 16.04.05 LTS,  4 cores in guest, 4 in host

using sumo yocto from variscite:
repo init -u https://github.com/varigit/variscite-bsp-platform.git -b refs/tags/sumo-fslc-4.9.88-mx6-v1.2
cd var-fslc-yocto
MACHINE=var-som-mx6 DISTRO=fslc-framebuffer . setup-environment build_fb
bitbake fsl-image-qt5

At first after ~3hours it suddenly happened, and then trying to login and continue, it immediately stopped again and again.
The computer also is very heated.

The next trial, I noticed that I used the virtual box with very low memory (200M) to I increased it to ~10G.
Yet on checking with "free" I still see that the free memory gets quite low sometimes ( to ~100M).

I also worked in a cooler environment (might be related too?)...
Yet, it did happen after ~4 hours, and then on login and bitbake again it continued till it finished.

Thanks,
Comment 16 ran 2018-12-19 22:13:47 UTC
The "next trial" which I mentioned before, was done after removing the previous build and doing everything from start,

Thanks.
Comment 17 ran 2018-12-19 22:20:18 UTC
One last comment:
If someone tries to reproduce it, I suggest to try to use virtualbox with low memory and see if it happens.
Comment 18 ran 2018-12-20 09:36:18 UTC
It seems like a temperature issue, not ram issue. Thank you.
Comment 20 Randy MacLeod 2022-02-17 16:11:48 UTC
This hasn't been seen in ages. If it's still a problem, please re-open.