| Summary: | Autobuilder built beagleboard 20130724-2 image failed | ||
|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Ning Zhang <ning.zhang> |
| Component: | general | Assignee: | Stefan Stanacar <sstncr> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | corneliux.stoicescu, elizabeth.flanagan, infras.ab.watcher, Infras.watcher, ke.zou, sgw, sstncr, yi.zhao |
| Version: | 1.5 | ||
| Target Milestone: | 1.5 M3 | ||
| Hardware: | Beagleboard | ||
| OS: | arm | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Ning Zhang
2013-07-25 07:02:56 UTC
So this is either a bitbake issue or the new sanity testing. Let me give a run down of what happened. The Sanity testing run failed and appears to have hung the bitbake process due to timing out. The fact that the bitbake process seems to have hung is somewhat surprising. When it tried to build arm/beagleboard/etc, the bitbake process still had a lock which caused the failure. So, I've bumped sanity timeout to 14800 seconds (which should be more than enough). However, it's concerning that either sanity is taking a REALLY long time or we had a bitbake lock. My initial guess is that the sanity tests took a long time and my bumping up the timeout will have fixed this. "ERROR: Only one copy of bitbake should be run against a build directory " - This is another effect of #4750 - ie bitbake doesn't kills it child processes in some cases. I know that Valentin has found a fix for that (there is a missing "import signal" in bitbake-worker AFAIR). I'm pretty sure that anything > 5000 is more than enough for any load. So we have another problem here, that leads to the testimage task not finishing (which I'll try to reproduce locally) so autobuilder kills the bitbake process but because (or an effect) of #4750 not all bitbake childs are killed hence all builds on the same builddir fail. Beth, while everything I said in the previous comment it's still valid, in this particular build we have the old timeout (which indeed might not be enough): command timed out: 1200 seconds without output, attempting to kill. If we have a build with 4800 timeout that failed I need to reproduce that to see what happens. I've bumped up to 14800, which should be *more* than enough. If we have a fix for #4750 then we should be fine. So this has happened again: http://autobuilder.yoctoproject.org:8011/builders/nightly-arm/builds/229, so there is clearly something wrong here (and now it's clearly not a timeout issue) Interesting that it only happened on qemuarm and that it didn't happen on the previous build http://autobuilder.yoctoproject.org:8011/builders/nightly-arm/builds/229 I've tried to reproduce this manually but I haven't had any luck (both on master and MUT). I'll keep looking into, I have a hunch that is somehow related to qemu serial connection and the read method we use... The fix for #4750 will only help properly killing the bitbake processes (so the next steps shouldn't fail anymore) but it won't help here. The way we read data from the serial console was unreliable and blocking (AutoBuilder seems to hit that often), so I've sent a patch on the ML changing the serial console type from unix socket to tcp and reverse the connection - don't let qemu act as server (wait for a connection). So now the serial console is used to save the boot log and make sure that we reached the login prompt. This should solve problems like http://autobuilder.yoctoproject.org:8011/builders/nightly-arm/builds/229/steps/Running%20Sanity%20Tests/logs/stdio (where the task doesn't finish). http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=0ba78c1162bb125850a0ee504ca6fbe5bf21247f Last nightly AB run had 2 sanity fails unrelated to the block. |