| Summary: | interrupting bitbake can leave zombie process | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Aaron Zinghini <aaron.zinghini> |
| Component: | bitbake | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED OBSOLETE | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | benjamin.esquivel, joshuagloe, poky.bs.watcher, poky.watcher, randy.macleod |
| Version: | 2.2.1 | ||
| Target Milestone: | 2.4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
How's does Jenkins abort the build? If you try and cancel an in-progress build with a keyboard interrupt/SIGINT bitbake will run active tasks to their conclusion and clean up after itself. If you stop the build with a larger hammer like SIGKILL, or if you trigger 3 keyboard interrupts/SIGINT, the shutdown is immediate and the cleanup doesn't happen. This often leaves your build tree in a broken state (resulting in errors like "ERROR: Only one copy of bitbake should be run against a build directory"). That being said, zombie processes shouldn't be left around in either case. Could you provide complete steps to reproduce your setup so that we might try and investigate? To my knowledge Jenkins sends a SIGTERM: https://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build The actual steps to reproduce this is kind of tricky and seems to be related to timing of when you abort, as most of the time it does not cause a problem. Eg. I was able to abort the job fine when it had got past the "Parsing recipes" and onto compiling, where as recently a colleague was able to cause the problem when they aborted during the recipe parsing stage. I've seen runaway/zombie processes coming out of bitbake not only with sigterm but with exceptions as well. It depends on the timing the exceptions/signals happen. I believe this needs a different testing approach of inducing signals/exceptions at various bitbake stages. We've been unable to get a handle on this issue to debug and fix it and the code handling the server has now been totally reworked. Not sure there is much we can do about this so closing as obsolete by the new server changes. If there is a way to reproduce this please reopen and we can reconsider. |
We have yocto building under Jenkins and have come across a problem where aborting a build can leave a zombie bitbake process lying around, which then causes subsequent builds to fail due to the error: ERROR: Only one copy of bitbake should be run against a build directory The zombie process: jenkins 24454 0.1 0.7 640264 256008 ? Sl 10:00 0:10 python3 /build/valleyisland/os/bitbake/bin/bitbake image-x11 The cooker log (under tmp/log/cooker/intel-corei7-64) shows only this: NOTE: Your conf/bblayers.conf has been automatically updated. NOTE: Your conf/bblayers.conf has been automatically updated. ERROR: UI received SIGTERM ERROR: UI received SIGTERM WARNING: Cooker received SIGHUP, shutting down... ERROR: UI received SIGTERM And the console log from Jenkins shows: ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-sato meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' NOTE: Your conf/bblayers.conf has been automatically updated. NOTE: Your conf/bblayers.conf has been automatically updated. Parsing recipes...ERROR: UI received SIGTERM ERROR: UI received SIGTERM Build was aborted Thanks