Bug 6934

Summary: bitbake hangs when PR server is down or invalid
Product: [Build System, Metadata & Runtime] BitBake Reporter: Ben Shelton <ben.shelton>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: VERIFIED FIXED QA Contact: Costin <costin.c.constantin>
Severity: major    
Priority: Medium+ CC: bogdanx.a.voiculescu, ke.zou, poky.bs.watcher, poky.watcher, yi.zhao, zhangle.yang
Version: 1.7   
Target Milestone: 1.8 M2   
Hardware: x86   
OS: Multiple   
Whiteboard: TC needed
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Ben Shelton 2014-11-05 22:19:12 UTC 
1) Get the latest bitbake and oe-core master as described in http://www.openembedded.org/wiki/OE-Core_Standalone_Setup (note that this also reproduces with Daisy)

2) Edit <build directory>/conf/local.conf and add the line:

PRSERV_HOST = "bob:8585"

3) Set up the environment and bitbake a package, e.g. `bitbake busybox`

You'll see a traceback like this:

Traceback (most recent call last):
  File "/ssd/builds/oe/oe-core/bitbake/bin/bitbake", line 382, in <module>
    ret = main()
  File "/ssd/builds/oe/oe-core/bitbake/bin/bitbake", line 347, in main
    server_connection = server.establishConnection(featureset)
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/server/process.py", line 247, in establishConnection
    raise BaseException(error)
BaseException: Traceback (most recent call last):
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/command.py", line 71, in runCommand
    result = command_method(self, commandline)
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/command.py", line 268, in setFeatures
    command.cooker.setFeatures(features)
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/cooker.py", line 164, in setFeatures
    raise Exception("Illegal state for feature set change")
Exception: Illegal state for feature set change

ERROR: Unable to set the cooker to the correct featureset: Traceback (most recent call last):
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/command.py", line 71, in runCommand
    result = command_method(self, commandline)
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/command.py", line 268, in setFeatures
    command.cooker.setFeatures(features)
  File "/ssd/builds/oe/oe-core/bitbake/lib/bb/cooker.py", line 164, in setFeatures
    raise Exception("Illegal state for feature set change")
Exception: Illegal state for feature set change

The build will stay in this state indefinitely.

4) Give Bitbake a Ctrl-C.  Now, do `ps -ef | grep bitbake`, and you'll see something like this:

bshelton 13109     1 99 16:07 pts/26   00:02:26 python /ssd/builds/oe/oe-core/bitbake/bin/bitbake busybox

Now do `cat /proc/<PID>/status` several times and check the VmSize line each time.  bitbake's memory usage continues to increase until it is killed.  However, the process sticks around unless you do 'kill -9' on it.

Ideally, in this case, I'd like to see an error message indicating that the PR server is down or not responding, and I'd like the build to exit cleanly.
Comment 2 Yi Zhao 2014-12-17 06:29:41 UTC 
I tested it with same steps in latest master (8657c2fd57600e0b0eaf2a14692de7453c78e1bb). The traceback disappeared. But the bitbake still hung and didn't exit.
################
$ bitbake busybox
ERROR: PRservice bob:8585 not available

Summary: There was 1 ERROR message shown, returning a non-zero exit code.

################
  
###############
$ ps aux|grep bitbake
zhaoyi    6803  0.4  0.6  97488 25536 pts/12   S+   14:20   0:01 python /home/zhaoyi/work/poky/poky-build/bitbake/bin/bitbake busybox
zhaoyi    6805  0.0  0.5 175644 22924 pts/12   S+   14:20   0:00 python /home/zhaoyi/work/poky/poky-build/bitbake/bin/bitbake busybox
zhaoyi    6846 26.8  8.9 508392 354556 pts/12  Sl+  14:20   1:26 python /home/zhaoyi/work/poky/poky-build/bitbake/bin/bitbake busybox
zhaoyi    6847 26.7  8.7 504612 349696 pts/12  Sl+  14:20   1:26 python /home/zhaoyi/work/poky/poky-build/bitbake/bin/bitbake busybox
################

And I can not terminate it by Ctrl-C .
Comment 3 Richard Purdie 2015-01-08 09:43:43 UTC 
Really fixed with: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1393e224b51d07297f615cc53dab4ef3bab72770

What happened is I added the broken line to local.conf and then fixed bitbake until it worked. I did not dirty the cache again and test a reparse with that broken line though. This time I've done that!
Comment 4 Costin 2015-03-12 10:55:23 UTC 
Hello,

Verified on latest master and it works:

poky hash: 0aff27522d467ee5d02289594207ddef4add7081

cosscat@ubnt:/media/yp_builds/poky/build$ bitbake busybox
ERROR: PRservice costin:8585 not available

Summary: There was 1 ERROR message shown, returning a non-zero exit code.
cosscat@ubnt:/media/yp_builds/poky/build$ echo $?
1

Bitbake exits without problem.

Costin