| Summary: | Bitbake server intermittent timeout | ||
|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Alexandre Belloni <alexandre.belloni> |
| Component: | oe-selftest | Assignee: | Unassigned <unassigned> |
| Status: | RESOLVED WORKSFORME | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alejandro, alexandre.belloni, anibal, anuj.mittal, chee.yang.lee, jon.mason, kory.maincent, leonardo.sandoval, liezhi.yang, randy.macleod, richard.purdie, steve, tvgamblin, valery.chernous |
| Version: | 3.3 | ||
| Target Milestone: | 4.2 M2 | ||
| Hardware: | x86 | ||
| OS: | x86_64 | ||
| Whiteboard: | AB-INT AB-FREQUENT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Alexandre Belloni
2021-01-26 21:22:38 UTC
2021-01-20 04:24:52,032 - oe-selftest - INFO - ======================================================================
2021-01-20 04:24:52,032 - oe-selftest - INFO - FAIL: devtool.DevtoolAddTests.test_devtool_add_fetch (subunit.RemotedTestCase)
2021-01-20 04:24:52,032 - oe-selftest - INFO - ----------------------------------------------------------------------
2021-01-20 04:24:52,032 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 434, in test_devtool_add_fetch
recipefile = get_bb_var('FILE', testrecipe)
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 284, in get_bb_var
return get_bb_vars([var], target, postconfig)[var]
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 246, in get_bb_vars
bbenv = get_bb_env(target, postconfig=postconfig)
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 240, in get_bb_env
return bitbake("-e %s" % target, postconfig=postconfig).output
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 232, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 210, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -e python-markupsafe' returned non-zero exit status 1:
Timeout while waiting for a reply from the bitbake server (60s)
2021-01-20 04:24:52,032 - oe-selftest - INFO - ======================================================================
2021-01-20 04:24:52,032 - oe-selftest - INFO - FAIL: devtool.DevtoolModifyTests.test_devtool_buildclean (subunit.RemotedTestCase)
2021-01-20 04:24:52,032 - oe-selftest - INFO - ----------------------------------------------------------------------
2021-01-20 04:24:52,032 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 630, in test_devtool_buildclean
bitbake('mdadm m4 -c buildclean')
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 232, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 210, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake mdadm m4 -c buildclean' returned non-zero exit status 1:
Timeout while waiting for a reply from the bitbake server (60s)
Happened again in: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1776 *** Bug 14225 has been marked as a duplicate of this bug. *** A new occurence here, right after setting up the environment: https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3061/steps/16/logs/stdio ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-full-cmdline core-image-sato core-image-weston meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' Other commonly useful commands are: - 'devtool' and 'recipetool' handle common recipe tasks - 'bitbake-layers' handles common layer tasks - 'oe-pkgdata-util' handles common target package tasks Timeout while waiting for a reply from the bitbake server (60s) Another timeout: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1850/steps/14/logs/stdio In our internal builds when we are building poky images (but oddly enough, NEVER when we are building WR images!), we see:
[09:15:24-0800] bitbake core-image-sato
[09:16:26-0800] Timeout while waiting for a reply from the bitbake server (60s)
It's really infrequent but happens often enough that it annoys me. From the last 4 months this error has happend 15-38 times/month:
19 10/2020
15 11/2020
22 12/2020
38 1/2021
24 2/2021
Of course we do 15K builds/month so it's never been a high priority.
At some point, we'll see if Sakib's periodic monitor script using flock can help us to understand what's going on there.
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1931/steps/21/logs/stdio This one is alose pretty early: NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 NOTE: No reply from server in 30s NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) Same in devtool tests. https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1937/steps/14/logs/stdio AssertionError: Command 'devtool reset cdrtools-native' returned non-zero exit status 1: NOTE: Starting bitbake server... NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) *** Bug 14278 has been marked as a duplicate of this bug. *** Happened again here during oe-selftest: AssertionError: Command 'devtool add http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2' returned non-zero exit status 1: NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 NOTE: No reply from server in 30s NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1984/steps/14/logs/stdio During oe-selftest on hardknott: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2042 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2036 master-next on centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/3358/steps/16/logs/stdio master-next on fedora33-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2052/steps/14/logs/stdio master-next on fedora33-ty-1 Trevor is looking at the bitbake server timeout issue that we see in WR Linux so adding him to CC. oe-selftest-ubuntu ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2108/steps/14/logs/stdio timeout while test for recipetool.RecipetoolTests.test_recipetool_create NOTE: Tasks Summary: Attempted 83 tasks of which 81 didn't need to be rerun and all succeeded. NOTE: No reply from server in 30s INFO: Source extracted to /tmp/recipetoolqahflxdy4o/srctree INFO: Recipe /tmp/recipetoolqahflxdy4o/logrotate_3.12.3.bb has been created; further editing may be required to make it fully functional NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) *** Bug 14390 has been marked as a duplicate of this bug. *** Reoccurance while testing GCC11 patches https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2059/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2149/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2182/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2188/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 FYI: The poky build timeout in the WR Linux build that I mentioned earlier seems to have stopped when we updated the version of docker that we used. It's possible that it isn't the version but rather a system uptime but I'll try to remember to comment if the problem shows up again. It was happening every day or two and it's been 13 days now so it's most likely gone. https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1481/steps/14/logs/stdio NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... oe-selftest fedora31-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2221 oe-selftest-fedora fedora33-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2245/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2247/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2249/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/39/builds/3617/steps/6/logs/stdio check-layer ubuntu1604-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/1613/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/1613/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/1613/steps/15/logs/stdio meta-intel debian9-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2271/steps/15/logs/stdio oe-selftest-centos centos7-ty-4 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3657/steps/14/logs/stdio qemuarm64 tumbleweed-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/119/builds/462/steps/10/logs/stdio reproducible-ubuntu ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3762/steps/16/logs/stdio qemumips64 fedora33-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2385/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2348/steps/14/logs/stdio oe-selftest-fedora fedora33-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1560/steps/14/logs/stdio oe-selftest ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1565/steps/14/logs/stdio oe-selftest centos8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4173/steps/19/logs/stdio ubuntu1604-ty-1 qemuppc https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2828/steps/14/logs/stdio ubuntu1804-ty-3 oe-selftest-ubuntu https://autobuilder.yoctoproject.org/typhoon/#builders/87/builds/2900/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-2 *** Bug 14662 has been marked as a duplicate of this bug. *** https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4549/steps/6/logs/stdio meta-mingw ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1713/steps/14/logs/stdio oe-selftest opensuse154-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4525/steps/6/logs/stdio meta-mingw ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3139/steps/14/logs/stdio ubuntu1804-ty-3 non-SSD There appears to be another instance of this https://autobuilder.yoctoproject.org/typhoon/#/builders/103/builds/3818 Bug 14767 has been opened for that last instance https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4955/steps/6/logs/stdio ubuntu1804-ty-3 164821914 updateConfig took ~90s Some timeout issues have been fixed but let's wait a while to see if this one is gone. https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3456/steps/15/logs/stdio │ debian9-ty-2 │ Non-SSD │ https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3491/steps/14/logs/stdio oe-selftest-ubuntu on ubuntu1804-ty-3 kirkstone, meta-aws, centos7-ty-4: NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/1220 https://autobuilder.yoctoproject.org/typhoon/#builders/79/builds/3716/steps/15/logs/stdio oe-selftest-centos centos7-ty-4 https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5429/steps/13/logs/stdio genericx86-64 stream8-ty-1 dunfell https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3817 oe-selftest-centos alma8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/5889/steps/26/logs/stdio multilib tumbleweed-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2081/steps/15/logs/stdio oe-selftest stream8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/15/builds/6772/steps/16/logs/stdio poky-tiny ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6481/steps/13/logs/stdio qemuarm64 ubuntu1804-ty-3 (same a-full) https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6409/steps/17/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6409/steps/20/logs/stdio qemuarm64 opensuse154-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4648/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4610/steps/15/logs/stdio oe-selftest-centos alma8-ty-2 Closing as bitbake changed dramatically, tracking new occurrences in a new bug. |