This is against the latest bitbake (master). Ever so often we will see some of our builds fail with the following output: 21:21:55 NOTE: Running task 6879 of 12679 (/scratch/jenkins_builds/poky-master-next/build/sources/oe-core/meta/recipes-core/volatile-binds/volatile-binds.bb:do_populate_sysroot) 21:21:55 NOTE: recipe volatile-binds-1.0-r0: task do_populate_sysroot: Started 21:21:57 NOTE: recipe run-postinsts-1.0-r0: task do_populate_sysroot: Succeeded 21:23:28 NOTE: No reply from server in 30s (for command ping at 02:23:28.132224) 21:23:58 Timeout while waiting for a reply from the bitbake server (60s at 02:23:58.225026) Once a bitbake fails, our build script tries to run the bitbake command a second time to try and get around other kinds of spurious recipe/networking errors. But then it just goes nowhere. 21:24:28 INFO: bitbake core-image-minimal build failed 1 times, retrying... 21:24:28 MACHINE=beaglebone-ai64 "bitbake" "-k" "core-image-minimal" 21:25:30 NOTE: Reconnecting to bitbake server... 21:25:30 NOTE: No reply from server in 30s (for command setFeatures at 02:25:00.167062) 21:25:30 NOTE: Retrying server connection (#1)... (02:25:30.200175) 21:26:30 NOTE: Reconnecting to bitbake server... 21:26:30 NOTE: No reply from server in 30s (for command setFeatures at 02:26:00.246052) 21:26:30 NOTE: Retrying server connection (#2)... (02:26:30.277327) 21:27:30 NOTE: Reconnecting to bitbake server... 21:27:30 NOTE: No reply from server in 30s (for command setFeatures at 02:27:00.323060) 21:27:30 NOTE: Retrying server connection (#3)... (02:27:30.353171) 21:27:30 NOTE: Reconnecting to bitbake server... 21:27:30 NOTE: Retrying server connection (#4)... (Traceback (most recent call last): 21:27:30 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake 21:27:30 server_connection = bb.server.process.connectProcessServer(sockname, featureset) 21:27:30 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer 21:27:30 sock.connect(os.path.basename(sockname)) 21:27:30 BlockingIOError: [Errno 11] Resource temporarily unavailable 21:27:30 , 02:27:30.354526) 21:27:35 NOTE: Reconnecting to bitbake server... 21:27:35 NOTE: Retrying server connection (#5)... (Traceback (most recent call last): 21:27:35 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake 21:27:35 server_connection = bb.server.process.connectProcessServer(sockname, featureset) 21:27:35 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer 21:27:35 sock.connect(os.path.basename(sockname)) 21:27:35 BlockingIOError: [Errno 11] Resource temporarily unavailable 21:27:35 , 02:27:35.370320) 21:27:40 NOTE: Reconnecting to bitbake server... 21:27:40 NOTE: Retrying server connection (#6)... (Traceback (most recent call last): 21:27:40 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake 21:27:40 server_connection = bb.server.process.connectProcessServer(sockname, featureset) 21:27:40 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer 21:27:40 sock.connect(os.path.basename(sockname)) 21:27:40 BlockingIOError: [Errno 11] Resource temporarily unavailable 21:27:40 , 02:27:40.375517) 21:27:45 NOTE: Reconnecting to bitbake server... 21:27:45 NOTE: Retrying server connection (#7)... (Traceback (most recent call last): 21:27:45 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/main.py", line 491, in setup_bitbake 21:27:45 server_connection = bb.server.process.connectProcessServer(sockname, featureset) 21:27:45 File "/scratch/jenkins_builds/poky-master-next/build/sources/bitbake/lib/bb/server/process.py", line 666, in connectProcessServer 21:27:45 sock.connect(os.path.basename(sockname)) 21:27:45 BlockingIOError: [Errno 11] Resource temporarily unavailable 21:27:45 , 02:27:45.385815) The last output was from 9:27pm, and I'm capturing this output at 9:59am the next morning. There is a Cooker process running at 100% and it will be hung there forever until we kill it. While I would love to fix whatever issue is causing the above failures, at least not hanging and leaving processes running amok should hopefully be achievable. Not sure the best way to detect and fail out.
Also collect the cooker daemon log next time.
I'm still waiting for this situation to occur again so that I can grab the logs. Not sure if you want to close this and I can reopen it once I get the requested logs...
We're a patient group! Let's give it another month or so. I've cleared the Whiteboard and set the Status to Accepted so that you won't get Needinfo pings.
Created attachment 5141 [details] Output log and cooker daemon log from 2025/09/12 Finally occurred again. Here is a tarball with the latest output.log (last several lines) and the requested cookerdaemon log file.
One log says: "NOTE: No reply from server in 30s (for command ping at 02:31:36.695323)" and the other has the last time stamped entry as: 2025-09-12 02:05:37 (330 MB/s) - ‘/scratch/jenkins_builds/poky-master-next/build/downloads/uninative/4c03d1ed2b7b4e823aca4a1a23d8f2e322f1770fc10e859adcede5777aff4f3a/x86_64-nativesdk-libc-4.9.tar.xz.tmp’ saved [7164424/7164424] so no logs anywhere close to the 02:31 timestamp, just 25 minutes before. This makes it all the more puzzling :/
I haven't seen this issue occur since moving to python 3.11 as suggested by Richard. There must have been something fixed that was causing this issue. Closing this item.