14201 2021-01-26 21:22:38 +0000 Bitbake server intermittent timeout 2023-01-12 15:52:28 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest 3.3 x86 x86_64 RESOLVED WORKSFORME AB-INT AB-FREQUENT Medium+ normal 4.2 M2 1 alexandre.belloni unassigned alejandro alexandre.belloni anibal anuj.mittal chee.yang.lee jon.mason kory.maincent leonardo.sandoval liezhi.yang randy.macleod richard.purdie steve tvgamblin valery.chernous --- --- No (bug/feature does not impact docs) oldest_to_newest 89036 0 alexandre.belloni 2021-01-26 21:22:38 +0000 autobuilder failure: 2021-01-20 04:25:01,252 - oe-selftest - INFO - RESULTS - devtool.DevtoolAddTests.test_devtool_add_fetch: FAILED (1369.50s) 2021-01-20 04:25:01,252 - oe-selftest - INFO - RESULTS - devtool.DevtoolModifyTests.test_devtool_buildclean: FAILED (2264.95s) Full log: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1739/steps/14/logs/stdio 89060 1 richard.purdie 2021-01-28 15:38:34 +0000 2021-01-20 04:24:52,032 - oe-selftest - INFO - ====================================================================== 2021-01-20 04:24:52,032 - oe-selftest - INFO - FAIL: devtool.DevtoolAddTests.test_devtool_add_fetch (subunit.RemotedTestCase) 2021-01-20 04:24:52,032 - oe-selftest - INFO - ---------------------------------------------------------------------- 2021-01-20 04:24:52,032 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 434, in test_devtool_add_fetch recipefile = get_bb_var('FILE', testrecipe) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 284, in get_bb_var return get_bb_vars([var], target, postconfig)[var] File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 246, in get_bb_vars bbenv = get_bb_env(target, postconfig=postconfig) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 240, in get_bb_env return bitbake("-e %s" % target, postconfig=postconfig).output File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 232, in bitbake return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 210, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) AssertionError: Command 'bitbake -e python-markupsafe' returned non-zero exit status 1: Timeout while waiting for a reply from the bitbake server (60s) 2021-01-20 04:24:52,032 - oe-selftest - INFO - ====================================================================== 2021-01-20 04:24:52,032 - oe-selftest - INFO - FAIL: devtool.DevtoolModifyTests.test_devtool_buildclean (subunit.RemotedTestCase) 2021-01-20 04:24:52,032 - oe-selftest - INFO - ---------------------------------------------------------------------- 2021-01-20 04:24:52,032 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 630, in test_devtool_buildclean bitbake('mdadm m4 -c buildclean') File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 232, in bitbake return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 210, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) AssertionError: Command 'bitbake mdadm m4 -c buildclean' returned non-zero exit status 1: Timeout while waiting for a reply from the bitbake server (60s) 89093 2 richard.purdie 2021-01-30 08:56:19 +0000 Happened again in: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1776 89223 3 randy.macleod 2021-02-11 15:49:22 +0000 *** Bug 14225 has been marked as a duplicate of this bug. *** 89343 4 alexandre.belloni 2021-02-23 00:32:44 +0000 A new occurence here, right after setting up the environment: https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3061/steps/16/logs/stdio ### Shell environment set up for builds. ### You can now run 'bitbake <target>' Common targets are: core-image-minimal core-image-full-cmdline core-image-sato core-image-weston meta-toolchain meta-ide-support You can also run generated qemu images with a command like 'runqemu qemux86' Other commonly useful commands are: - 'devtool' and 'recipetool' handle common recipe tasks - 'bitbake-layers' handles common layer tasks - 'oe-pkgdata-util' handles common target package tasks Timeout while waiting for a reply from the bitbake server (60s) 89419 5 alexandre.belloni 2021-02-26 01:55:54 +0000 Another timeout: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1850/steps/14/logs/stdio 89423 6 randy.macleod 2021-02-26 16:33:42 +0000 In our internal builds when we are building poky images (but oddly enough, NEVER when we are building WR images!), we see: [09:15:24-0800] bitbake core-image-sato [09:16:26-0800] Timeout while waiting for a reply from the bitbake server (60s) It's really infrequent but happens often enough that it annoys me. From the last 4 months this error has happend 15-38 times/month: 19 10/2020 15 11/2020 22 12/2020 38 1/2021 24 2/2021 Of course we do 15K builds/month so it's never been a high priority. At some point, we'll see if Sakib's periodic monitor script using flock can help us to understand what's going on there. 89442 7 alexandre.belloni 2021-02-27 18:16:45 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1837/steps/14/logs/stdio 89486 8 alexandre.belloni 2021-03-03 12:46:52 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1931/steps/21/logs/stdio This one is alose pretty early: NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 NOTE: No reply from server in 30s NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) 89533 9 kory.maincent 2021-03-09 10:35:36 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/3159/steps/6/logs/stdio 89553 10 kory.maincent 2021-03-09 16:14:49 +0000 Same in devtool tests. https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1937/steps/14/logs/stdio AssertionError: Command 'devtool reset cdrtools-native' returned non-zero exit status 1: NOTE: Starting bitbake server... NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) 89564 11 kory.maincent 2021-03-11 14:05:30 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/3192/steps/12/logs/stdio 89581 12 randy.macleod 2021-03-11 15:56:23 +0000 *** Bug 14278 has been marked as a duplicate of this bug. *** 89886 13 alejandro 2021-04-07 18:29:32 +0000 Happened again here during oe-selftest: AssertionError: Command 'devtool add http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2' returned non-zero exit status 1: NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 NOTE: No reply from server in 30s NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1984/steps/14/logs/stdio 90107 14 anuj.mittal 2021-04-26 00:43:19 +0000 During oe-selftest on hardknott: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2042 90208 15 alexandre.belloni 2021-05-01 22:41:58 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2036 master-next on centos8-ty-1 90257 16 alexandre.belloni 2021-05-06 11:35:14 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/3358/steps/16/logs/stdio master-next on fedora33-ty-1 90272 17 alexandre.belloni 2021-05-06 14:41:52 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2052/steps/14/logs/stdio master-next on fedora33-ty-1 90301 18 randy.macleod 2021-05-06 15:53:14 +0000 Trevor is looking at the bitbake server timeout issue that we see in WR Linux so adding him to CC. 90309 19 alexandre.belloni 2021-05-06 18:30:29 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/121/builds/41/steps/17/logs/stdio 90337 20 chee.yang.lee 2021-05-10 09:08:20 +0000 oe-selftest-ubuntu ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2108/steps/14/logs/stdio timeout while test for recipetool.RecipetoolTests.test_recipetool_create NOTE: Tasks Summary: Attempted 83 tasks of which 81 didn't need to be rerun and all succeeded. NOTE: No reply from server in 30s INFO: Source extracted to /tmp/recipetoolqahflxdy4o/srctree INFO: Recipe /tmp/recipetoolqahflxdy4o/logrotate_3.12.3.bb has been created; further editing may be required to make it fully functional NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) 90364 21 randy.macleod 2021-05-13 14:47:37 +0000 *** Bug 14390 has been marked as a duplicate of this bug. *** 90456 22 jon.mason 2021-05-21 12:52:56 +0000 Reoccurance while testing GCC11 patches https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2059/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 90632 23 alexandre.belloni 2021-06-16 00:30:33 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2149/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 90649 24 alexandre.belloni 2021-06-16 09:05:02 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2182/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3 90657 25 alexandre.belloni 2021-06-16 13:43:36 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2188/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 90662 26 randy.macleod 2021-06-17 00:14:40 +0000 FYI: The poky build timeout in the WR Linux build that I mentioned earlier seems to have stopped when we updated the version of docker that we used. It's possible that it isn't the version but rather a system uptime but I'll try to remember to comment if the problem shows up again. It was happening every day or two and it's been 13 days now so it's most likely gone. 90700 27 alexandre.belloni 2021-06-17 22:40:47 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1481/steps/14/logs/stdio NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... oe-selftest fedora31-ty-1 90801 28 alexandre.belloni 2021-06-30 20:43:32 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2221 oe-selftest-fedora fedora33-ty-1 90802 29 alexandre.belloni 2021-06-30 22:13:23 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2245/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 90805 30 alexandre.belloni 2021-06-30 22:25:04 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2247/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 90806 31 alexandre.belloni 2021-06-30 22:27:21 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2249/steps/14/logs/stdio oe-selftest-centos centos8-ty-1 90831 32 alexandre.belloni 2021-07-01 18:42:54 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/39/builds/3617/steps/6/logs/stdio check-layer ubuntu1604-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/1613/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/1613/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/1613/steps/15/logs/stdio meta-intel debian9-ty-2 90853 33 alexandre.belloni 2021-07-04 14:02:22 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2271/steps/15/logs/stdio oe-selftest-centos centos7-ty-4 90938 34 alexandre.belloni 2021-07-15 14:47:26 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3657/steps/14/logs/stdio qemuarm64 tumbleweed-ty-3 90979 35 alexandre.belloni 2021-07-22 12:53:35 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/119/builds/462/steps/10/logs/stdio reproducible-ubuntu ubuntu1804-ty-3 91143 36 alexandre.belloni 2021-08-11 10:55:28 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3762/steps/16/logs/stdio qemumips64 fedora33-ty-1 91148 37 alexandre.belloni 2021-08-11 15:18:25 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2385/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-2 91151 38 alexandre.belloni 2021-08-11 20:24:55 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2348/steps/14/logs/stdio oe-selftest-fedora fedora33-ty-1 91187 39 alexandre.belloni 2021-08-13 13:13:09 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1560/steps/14/logs/stdio oe-selftest ubuntu1804-ty-3 91239 40 alexandre.belloni 2021-08-19 18:49:55 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1565/steps/14/logs/stdio oe-selftest centos8-ty-1 91931 41 chee.yang.lee 2021-11-09 13:43:42 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4173/steps/19/logs/stdio ubuntu1604-ty-1 qemuppc 91936 42 chee.yang.lee 2021-11-10 01:26:48 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2828/steps/14/logs/stdio ubuntu1804-ty-3 oe-selftest-ubuntu 92117 43 alexandre.belloni 2021-12-09 10:54:50 +0000 https://autobuilder.yoctoproject.org/typhoon/#builders/87/builds/2900/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2004-ty-2 92179 44 randy.macleod 2021-12-16 15:44:50 +0000 *** Bug 14662 has been marked as a duplicate of this bug. *** 92238 45 alexandre.belloni 2022-01-06 18:46:16 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4549/steps/6/logs/stdio meta-mingw ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1713/steps/14/logs/stdio oe-selftest opensuse154-ty-1 92241 46 alexandre.belloni 2022-01-07 14:37:49 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4540 92253 47 alexandre.belloni 2022-01-07 16:33:36 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2996/steps/14/logs/stdio 92267 48 alexandre.belloni 2022-01-08 00:37:11 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4525/steps/6/logs/stdio meta-mingw ubuntu1804-ty-3 92335 49 alexandre.belloni 2022-01-17 15:21:13 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3027/steps/15/logs/stdio 92336 50 alexandre.belloni 2022-01-17 15:27:22 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3021/steps/15/logs/stdio 92497 51 alexandre.belloni 2022-02-09 22:52:39 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3139/steps/14/logs/stdio ubuntu1804-ty-3 non-SSD 92838 52 jon.mason 2022-03-23 20:40:49 +0000 There appears to be another instance of this https://autobuilder.yoctoproject.org/typhoon/#/builders/103/builds/3818 92853 53 alexandre.belloni 2022-03-24 15:09:50 +0000 Bug 14767 has been opened for that last instance 92891 54 alexandre.belloni 2022-03-31 13:41:07 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/89/builds/4955/steps/6/logs/stdio ubuntu1804-ty-3 164821914 updateConfig took ~90s 92979 55 randy.macleod 2022-04-11 14:56:59 +0000 Some timeout issues have been fixed but let's wait a while to see if this one is gone. 93146 56 leonardo.sandoval 2022-04-20 23:35:45 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3456/steps/15/logs/stdio │ debian9-ty-2 │ Non-SSD │ 93204 57 alexandre.belloni 2022-04-28 15:00:38 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3491/steps/14/logs/stdio oe-selftest-ubuntu on ubuntu1804-ty-3 93316 58 steve 2022-05-25 18:01:09 +0000 kirkstone, meta-aws, centos7-ty-4: NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 NOTE: No reply from server in 30s Timeout while waiting for a reply from the bitbake server (60s) https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/1220 93567 59 alexandre.belloni 2022-06-23 00:02:08 +0000 https://autobuilder.yoctoproject.org/typhoon/#builders/79/builds/3716/steps/15/logs/stdio oe-selftest-centos centos7-ty-4 93621 60 alexandre.belloni 2022-06-30 15:03:31 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5429/steps/13/logs/stdio genericx86-64 stream8-ty-1 93665 61 steve 2022-07-13 18:21:50 +0000 dunfell https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3817 oe-selftest-centos alma8-ty-1 94161 62 alexandre.belloni 2022-10-14 13:25:47 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/5889/steps/26/logs/stdio multilib tumbleweed-ty-3 94566 63 alexandre.belloni 2022-12-22 13:47:48 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2081/steps/15/logs/stdio oe-selftest stream8-ty-1 94665 64 alexandre.belloni 2023-01-11 21:39:31 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/15/builds/6772/steps/16/logs/stdio poky-tiny ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6481/steps/13/logs/stdio qemuarm64 ubuntu1804-ty-3 (same a-full) 94670 65 alexandre.belloni 2023-01-11 22:20:02 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6409/steps/17/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/6409/steps/20/logs/stdio qemuarm64 opensuse154-ty-3 94686 66 alexandre.belloni 2023-01-12 13:39:46 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4648/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1 94687 67 alexandre.belloni 2023-01-12 13:49:26 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4610/steps/15/logs/stdio oe-selftest-centos alma8-ty-2 94703 68 alexandre.belloni 2023-01-12 15:52:28 +0000 Closing as bitbake changed dramatically, tracking new occurrences in a new bug.