Bug 15955

Summary: No reply from server error handling
Product: [Build System, Metadata & Runtime] BitBake Reporter: Ryan Eatmon <reatmon>
Component: bitbakeAssignee: Ryan Eatmon <reatmon>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie
Version: unspecified   
Target Milestone: 5.3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
Output log and cooker daemon log from 2025/09/12 none

Description Ryan Eatmon 2025-08-15 15:12:47 UTC 
This is against the latest bitbake (master).


Ever so often we will see some of our builds fail with the following output:


21:21:55 NOTE: Running task 6879 of 12679 (/scratch/jenkins_builds/poky-master-next/build/sources/oe-core/meta/recipes-core/volatile-binds/volatile-binds.bb:do_populate_sysroot)
21:21:55 NOTE: recipe volatile-binds-1.0-r0: task do_populate_sysroot: Started
21:21:57 NOTE: recipe run-postinsts-1.0-r0: task do_populate_sysroot: Succeeded
21:23:28 NOTE: No reply from server in 30s (for command ping at 02:23:28.132224)
21:23:58 Timeout while waiting for a reply from the bitbake server (60s at 02:23:58.225026)

Once a bitbake fails, our build script tries to run the bitbake command a second time to try and get around other kinds of spurious recipe/networking errors.  But then it just goes nowhere.

21:24:28 INFO: bitbake core-image-minimal build failed 1 times, retrying...
21:24:28 MACHINE=beaglebone-ai64 "bitbake" "-k" "core-image-minimal"
21:25:30 NOTE: Reconnecting to bitbake server...
21:25:30 NOTE: No reply from server in 30s (for command setFeatures at 02:25:00.167062)
21:25:30 NOTE: Retrying server connection (#1)... (02:25:30.200175)
21:26:30 NOTE: Reconnecting to bitbake server...
21:26:30 NOTE: No reply from server in 30s (for command setFeatures at 02:26:00.246052)
21:26:30 NOTE: Retrying server connection (#2)... (02:26:30.277327)
21:27:30 NOTE: Reconnecting to bitbake server...
21:27:30 NOTE: No reply from server in 30s (for command setFeatures at 02:27:00.323060)
21:27:30 NOTE: Retrying server connection (#3)... (02:27:30.353171)
21:27:30 NOTE: Reconnecting to bitbake server...
21:27:30 NOTE: Retrying server connection (#4)... (Traceback (most recent call last):
21:27:30   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake
21:27:30     server_connection = bb.server.process.connectProcessServer(sockname, featureset)
21:27:30   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer
21:27:30     sock.connect(os.path.basename(sockname))
21:27:30 BlockingIOError: [Errno 11] Resource temporarily unavailable
21:27:30 , 02:27:30.354526)
21:27:35 NOTE: Reconnecting to bitbake server...
21:27:35 NOTE: Retrying server connection (#5)... (Traceback (most recent call last):
21:27:35   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake
21:27:35     server_connection = bb.server.process.connectProcessServer(sockname, featureset)
21:27:35   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer
21:27:35     sock.connect(os.path.basename(sockname))
21:27:35 BlockingIOError: [Errno 11] Resource temporarily unavailable
21:27:35 , 02:27:35.370320)
21:27:40 NOTE: Reconnecting to bitbake server...
21:27:40 NOTE: Retrying server connection (#6)... (Traceback (most recent call last):
21:27:40   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake
21:27:40     server_connection = bb.server.process.connectProcessServer(sockname, featureset)
21:27:40   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer
21:27:40     sock.connect(os.path.basename(sockname))
21:27:40 BlockingIOError: [Errno 11] Resource temporarily unavailable
21:27:40 , 02:27:40.375517)
21:27:45 NOTE: Reconnecting to bitbake server...
21:27:45 NOTE: Retrying server connection (#7)... (Traceback (most recent call last):
21:27:45   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake
21:27:45     server_connection = bb.server.process.connectProcessServer(sockname, featureset)
21:27:45   File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer
21:27:45     sock.connect(os.path.basename(sockname))
21:27:45 BlockingIOError: [Errno 11] Resource temporarily unavailable
21:27:45 , 02:27:45.385815)

The last output was from 9:27pm, and I'm capturing this output at 9:59am the next morning.  There is a Cooker process running at 100% and it will be hung there forever until we kill it.

While I would love to fix whatever issue is causing the above failures, at least not hanging and leaving processes running amok should hopefully be achievable.  Not sure the best way to detect and fail out.
Comment 1 Randy MacLeod 2025-08-21 14:38:59 UTC 
Also collect the cooker daemon log next time.
Comment 2 Ryan Eatmon 2025-09-09 19:39:13 UTC 
I'm still waiting for this situation to occur again so that I can grab the logs.  Not sure if you want to close this and I can reopen it once I get the requested logs...
Comment 3 Randy MacLeod 2025-09-09 19:48:40 UTC 
We're a patient group! Let's give it another month or so.
I've cleared the Whiteboard and set the Status to Accepted
so that you won't get Needinfo pings.
Comment 4 Ryan Eatmon 2025-09-12 14:29:00 UTC 
Created attachment 5141 [details]
Output log and cooker daemon log from 2025/09/12

Finally occurred again.  Here is a tarball with the latest output.log (last several lines) and the requested cookerdaemon log file.
Comment 5 Richard Purdie 2025-09-12 15:27:33 UTC 
One log says:

"NOTE: No reply from server in 30s (for command ping at 02:31:36.695323)"

and the other has the last time stamped entry as:

2025-09-12 02:05:37 (330 MB/s) - ‘/scratch/jenkins_builds/poky-master-next/build/downloads/uninative/4c03d1ed2b7b4e823aca4a1a23d8f2e322f1770fc10e859adcede5777aff4f3a/x86_64-nativesdk-libc-4.9.tar.xz.tmp’ saved [7164424/7164424]

so no logs anywhere close to the 02:31 timestamp, just 25 minutes before. This makes it all the more puzzling :/
Comment 6 Ryan Eatmon 2026-02-17 15:20:45 UTC 
I haven't seen this issue occur since moving to python 3.11 as suggested by Richard.  There must have been something fixed that was causing this issue.  Closing this item.