Bug 15237

Summary: AB-INT: Timeout while waiting for a reply from the bitbake server
Product: [QA/Testing] Functional (self) Testing Reporter: Luca Ceresoli <luca.ceresoli>
Component: oe-selftestAssignee: Chen Qi <Qi.Chen>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: alexandre.belloni, joaomarcos.costa, liezhi.yang, mathieu.dubois-briand, randy.macleod, richard.purdie
Version: unspecified   
Target Milestone: 5.2   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Luca Ceresoli 2023-10-13 20:53:21 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5897/steps/14/logs/stdio

oe-selftest-centos on alma9-ty-1

The stdio log shows:

  NOTE: No reply from server in 30s (for command ping at 12:23:12.994729)
  Timeout while waiting for a reply from the bitbake server (60s at 12:23:43.095454)

AAlso from the stdio log:

  AssertionError: Command 'bitbake  virtual/kernel' returned non-zero exit status 1:

Similar to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15043
but likely unrelated.

It would be useful to have the bitbake-cookerdaemon.log file, but for this instance it is gone.
Comment 2 Alexandre Belloni 2023-11-20 22:19:00 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/7509/steps/10/logs/stdio

qemuarm64-armhost ubuntu2204-arm-1
Comment 3 Alexandre Belloni 2023-11-20 22:38:28 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/7505/steps/10/logs/stdio

qemuarm64-armhost ubuntu2204-arm-1
Comment 4 Alexandre Belloni 2024-04-04 21:33:40 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6572/steps/14/logs/stdio

oe-selftest-centos alma9-ty-2

NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#8)... (Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/main.py", line 489, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 708, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
, 19:53:24.849271)
Comment 5 Alexandre Belloni 2024-04-05 21:54:29 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6391/steps/14/logs/stdio

oe-selftest-centos alma9-ty-2
Comment 6 João Marcos Costa 2024-07-29 12:31:25 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/6670/steps/11/logs/stdio

qemuarm64-ptest ubuntu2004-arm-1

The error messages in the bottom of the stdio may not be precisely the same, but it is still a timeout issue.
Comment 7 Alexandre Belloni 2024-09-06 14:32:14 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3816/steps/14/logs/stdio

oe-selftest-armhost ubuntu2204-arm-1
Comment 8 Mathieu Dubois-Briand 2024-09-12 08:29:50 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/8751/steps/10/logs/stdio

qemuarm64-armhost on ubuntu2204-arm-1
Comment 9 Mathieu Dubois-Briand 2024-09-24 14:35:32 UTC
oe-selftest-debian debian11-ty-1
Error talking to server: Connection closed
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6586/steps/11/logs/stdio

Timeout while waiting for a reply from the bitbake server (60s at 03:22:37.188037)
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/7022/steps/14/logs/stdio
Comment 11 Mathieu Dubois-Briand 2024-10-04 15:30:34 UTC
oe-selftest fedora38-ty-6
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2669/steps/14/logs/stdio
Comment 12 Mathieu Dubois-Briand 2024-10-14 13:05:25 UTC
oe-selftest-fedora fedora41-vk-1
https://valkyrie.yoctoproject.org//#/builders/48/builds/185/steps/14/logs/stdio

Richard saved some data: https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/

12:43 < RP> mathieudb: the key message in cookerdaemon is "Idle loop didn't finish queued commands after 30s, exiting."
12:43 < RP> in there twice
12:43  * RP is hoping there are timestamps in the json log
Comment 13 Mathieu Dubois-Briand 2024-10-15 06:08:51 UTC
oe-selftest-fedora fedora41-vk-1
https://valkyrie.yoctoproject.org/#/builders/48/builds/192

Thanks Richard, again:
https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/2/
Comment 14 Mathieu Dubois-Briand 2024-11-05 11:06:51 UTC
oe-selftest-fedora fedora41-vk-1
https://valkyrie.yoctoproject.org/#/builders/48/builds/279/steps/14/logs/stdio
Comment 15 Randy MacLeod 2025-05-15 18:35:21 UTC
Not seen in YP for ~6 months. Shall we declare this fixed ?

YP does have the new, faster cluster.

This is still seen in WR Linux, where we have:
 - a slow template bitbake patch and
 - we overload our container-based builds.
We have not had a customer report the issue.
Comment 16 Randy MacLeod 2025-06-05 15:04:09 UTC
There were some timeout-related commits but I can't find them...
WR still has some timeouts but that's their problem since they have a slow template feature and overloaded builders!