| Summary: | bitbake server timeout on first 'server.runCommand' call after build starts | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | David Reyna <david.reyna> | ||||
| Component: | bitbake | Assignee: | Richard Purdie <richard.purdie> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | major | ||||||
| Priority: | High | CC: | poky.bs.watcher, poky.watcher, randy.macleod | ||||
| Version: | 2.4 | ||||||
| Target Milestone: | 2.4 M3 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| See Also: | https://bugzilla.yoctoproject.org/show_bug.cgi?id=12116 | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
Created attachment 3947 [details]
unit test version of knotty to demonstrate defect
Attached is a simple replicator of the server timeout using knotty (to avoid the extra steps of setting up Toaster).
1) Place the attached "knotty_11898.py" into "bitbake/lib/bb/ui"
2) Observe the simple server query added under "bb.event.ParseProgress":
if isinstance(event, bb.event.ParseProgress):
### TRY TO GET VALUE FROM BB SERVER ###
print("vvv BEFORE SERVER.RUNCOMMAND() vvv")
tmp_dir = server.runCommand(["getVariable", "TMPDIR"])[0]
print("^^^ AFTER SERVER.RUNCOMMAND() ^^^")
This is an example of a simple query that Toaster has been making for years in processing the bb events.
3) Run these commands:
[poky]$ . ./oe-init-build-env
[build]$ rm -rf cache/* tmp/ sstate-control/
[build]$ bitbake -u knotty_11898 quilt-native
NOTE: Starting bitbake server...
WARNING: Host distribution "ubuntu-14.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
vvv BEFORE SERVER.RUNCOMMAND() vvv | ETA: --:--:--
Timeout while waiting for a reply from the bitbake server
[build]$
NOTE: I did not need to do the "rm -rf ..." to replicate this error, but you may need to.
(In reply to comment #1) > Created attachment 3947 [details] > unit test version of knotty to demonstrate defect > > Attached is a simple replicator of the server timeout using knotty (to avoid > the extra steps of setting up Toaster). > > 1) Place the attached "knotty_11898.py" into "bitbake/lib/bb/ui" > > 2) Observe the simple server query added under "bb.event.ParseProgress": > > if isinstance(event, bb.event.ParseProgress): > > ### TRY TO GET VALUE FROM BB SERVER ### > print("vvv BEFORE SERVER.RUNCOMMAND() vvv") > tmp_dir = server.runCommand(["getVariable", "TMPDIR"])[0] > print("^^^ AFTER SERVER.RUNCOMMAND() ^^^") > > This is an example of a simple query that Toaster has been making for years > in processing the bb events. > > 3) Run these commands: > > [poky]$ . ./oe-init-build-env > [build]$ rm -rf cache/* tmp/ sstate-control/ > [build]$ bitbake -u knotty_11898 quilt-native > NOTE: Starting bitbake server... > WARNING: Host distribution "ubuntu-14.04" has not been validated with this > version of the build system; you may possibly experience unexpected > failures. It is recommended that you use a tested distribution. > vvv BEFORE SERVER.RUNCOMMAND() vvv | ETA: > --:--:-- > Timeout while waiting for a reply from the bitbake server > [build]$ > > > NOTE: I did not need to do the "rm -rf ..." to replicate this error, but you > may need to. Thanks David, this really does help illustrate the problem. The issue appears to be multiple nested commands. I'll see what I can figure out about what is going on but this should speed things up. *** Bug 11833 has been marked as a duplicate of this bug. *** |
Recent changes in the bitbake server has caused a serious regression in Toaster UI when running builds from the command line. * The issue: 1) Normally Toaster: (a) makes a few server calls to fetch variable values (WORKS) (b) runs the server call to start the build, (WORKS) (c) does addition sever calls to fetch variables to process the incoming event stream (NOW FAILS) 2) Now, the first server call after the build starts the server will time out with the message "Timeout while waiting for a reply from the bitbake server" and the build fails. 3) Here is where the build is failing, in the "toasterui -> BuildInfoHelper() -> __init__": self.tmp_dir = self.server.runCommand(["getVariable", "TMPDIR"])[0] The timeout occurs from this call, but that just happens to be the first 'server.runCommand' after the build started (see below). * Reproducer 1) Set up Toaster $ bash $ . ./oe-init-build-env $ . toaster start noweb 2) Start build $ rm -rf cache/* tmp/ sstate-control/ $ bitbake core-image-base <<< 5 seconds ... >>> Timeout while waiting for a reply from the bitbake server $ 3) Repeat #2 to again trigger the error * What I have tried: 1) I have tested increasing the 5 second timeout to say 10 seconds, but it still fails. 2) I have tried moving the Toaster code around and commenting out server calls so that the first call is as late as possible (in case this is just build start-up overhead problem). I have specifically tested delaying any server access until after the "bb.event.TreeDataPreparationStarted" event (easily 30 seconds after the build started). It can sometimes work in this case - but only if I also increase the server timeout to 10 seconds. 3) If I "rm -rf cache/* tmp/ sstate-control/" before the build, I can consistently get it to fail. If I do not, a subsequent build attempt has a better chance of working. 4) On occasion the observed error is in the server retry code, and I see this: [build]$ bitbake core-image-base NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/opt/dreyna/yp-2.4/poky/bitbake/lib/bb/main.py", line 449, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/opt/dreyna/yp-2.4/poky/bitbake/lib/bb/server/process.py", line 445, in connectProcessServer server_connection.connection.updateFeatureSet(featureset) File "/opt/dreyna/yp-2.4/poky/bitbake/lib/bb/server/process.py", line 319, in updateFeatureSet _, error = self.runCommand(["setFeatures", featureset]) File "/opt/dreyna/yp-2.4/poky/bitbake/lib/bb/server/process.py", line 316, in runCommand return self.recv.get() File "/opt/dreyna/yp-2.4/poky/bitbake/lib/bb/server/process.py", line 559, in get res = self.reader.recv_bytes() File "/usr/lib/python3.4/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib/python3.4/multiprocessing/connection.py", line 416, in _recv_bytes buf = self._recv(4) File "/usr/lib/python3.4/multiprocessing/connection.py", line 389, in _recv raise EOFError EOFError ) ... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection... ERROR: Unable to connect to bitbake server, or start one [build]$ 5) I believe that this error is not visible with Knotty because that does not make any server requests after the build starts (outside of the build cancel request). The other UIs however might be similarly impacted. 6) If the build output makes it to the display of repos and commit tags, the build will succeed.