13911 2020-05-20 03:56:06 +0000 SystemExit failure when running oe-selftests 2020-07-23 08:33:45 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 3.1 x86 Multiple RESOLVED FIXED AB-INT Medium+ normal 3.2 M2 1 richard.purdie kai.kang poky.bs.watcher poky.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 87257 0 richard.purdie 2020-05-20 03:56:06 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/956 2020-05-19 13:46:43,487 - oe-selftest - INFO - 9: 2/19 142/406 (863.67s) (incompatible_lic.IncompatibleLicensePerImageTests.test_bash_and_license) 2020-05-19 13:47:40,171 - oe-selftest - INFO - incompatible_lic.IncompatibleLicensePerImageTests.test_bash_default (subunit.RemotedTestCase) 2020-05-19 13:47:40,171 - oe-selftest - INFO - ... ok 2020-05-19 13:47:42,344 - oe-selftest - INFO - 9: 3/19 143/406 (56.68s) (incompatible_lic.IncompatibleLicensePerImageTests.test_bash_default) NOTE: Starting bitbake server... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 2020-05-19 13:51:02,062 - oe-selftest - INFO - incompatible_lic.IncompatibleLicensePerImageTests.test_bash_or_license (subunit.RemotedTestCase) 2020-05-19 13:51:02,062 - oe-selftest - INFO - ... ok 2020-05-19 13:51:02,062 - oe-selftest - INFO - 9: 4/19 144/406 (200.90s) (incompatible_lic.IncompatibleLicensePerImageTests.test_bash_or_license) ERROR: Unable to start bitbake server (False) ERROR: Server log for this session (/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-5872/bitbake-cookerdaemon.log): --- Starting bitbake server pid 48545 at 2020-05-19 13:49:24.295548 --- NOTE: Reconnecting to bitbake server... 2020-05-19 13:51:17,722 - oe-selftest - INFO - tinfoil.TinfoilTests.test_datastore_operations (subunit.RemotedTestCase) 2020-05-19 13:51:27,732 - oe-selftest - INFO - ... ERROR 2020-05-19 13:51:27,899 - oe-selftest - INFO - 10: 49/58 145/406 (278.26s) (tinfoil.TinfoilTests.test_datastore_operations) 2020-05-19 13:51:27,899 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/tinfoil.py", line 145, in test_datastore_operations tinfoil.prepare(config_only=True) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/tinfoil.py", line 391, in prepare self.server_connection, ui_module = setup_bitbake(config_params, File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/main.py", line 438, in setup_bitbake server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 447, in __init__ raise SystemExit(1) SystemExit: 1 NOTE: Retrying server connection (#1)... NOTE: Bitbake server didn't start within 5 seconds, waiting for 90 ERROR: Unable to start bitbake server (False) ERROR: Server log for this session (/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-5872/bitbake-cookerdaemon.log): --- Starting bitbake server pid 48545 at 2020-05-19 13:49:24.295548 --- NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#2)... NOTE: Starting bitbake server... 2020-05-19 13:52:33,890 - oe-selftest - INFO - tinfoil.TinfoilTests.test_expand (subunit.RemotedTestCase) 2020-05-19 13:52:33,890 - oe-selftest - INFO - ... ok 2020-05-19 13:52:33,890 - oe-selftest - INFO - 10: 50/58 146/406 (76.26s) (tinfoil.TinfoilTests.test_expand) NOTE: Starting bitbake server... 2020-05-19 13:52:36,057 - oe-selftest - INFO - tinfoil.TinfoilTests.test_getvar (subunit.RemotedTestCase) NOTE: Reconnecting to bitbake server... 2020-05-19 13:52:36,673 - oe-selftest - INFO - ... ok NOTE: Previous bitbake instance shutting down?, waiting to retry... 2020-05-19 13:52:36,673 - oe-selftest - INFO - 10: 51/58 147/406 (2.17s) (tinfoil.TinfoilTests.test_getvar) NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#1)... NOTE: Starting bitbake server... 2020-05-19 13:52:39,002 - oe-selftest - INFO - tinfoil.TinfoilTests.test_getvar_bb_origenv (subunit.RemotedTestCase) 2020-05-19 13:52:39,003 - oe-selftest - INFO - ... ok 2020-05-19 13:52:39,003 - oe-selftest - INFO - 10: 52/58 148/406 (2.95s) (tinfoil.TinfoilTests.test_getvar_bb_origenv) 2020-05-19 13:53:45,296 - oe-selftest - INFO - tinfoil.TinfoilTests.test_list_recipes (subunit.RemotedTestCase) 2020-05-19 13:53:45,297 - oe-selftest - INFO - ... ok 2020-05-19 13:53:45,297 - oe-selftest - INFO - 10: 53/58 149/406 (66.29s) (tinfoil.TinfoilTests.test_list_recipes) 2020-05-19 13:53:52,226 - oe-selftest - INFO - tinfoil.TinfoilTests.test_parse_recipe (subunit.RemotedTestCase) 2020-05-19 13:53:52,226 - oe-selftest - INFO - ... ok 2020-05-19 13:53:52,227 - oe-selftest - INFO - 10: 54/58 150/406 (6.93s) (tinfoil.TinfoilTests.test_parse_recipe) 2020-05-19 13:53:54,744 - oe-selftest - INFO - tinfoil.TinfoilTests.test_parse_recipe_copy_expand (subunit.RemotedTestCase) NOTE: Reconnecting to bitbake server... 2020-05-19 13:53:55,379 - oe-selftest - INFO - ... ok 2020-05-19 13:53:55,379 - oe-selftest - INFO - 10: 55/58 151/406 (2.52s) (tinfoil.TinfoilTests.test_parse_recipe_copy_expand) NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... 87289 1 randy.macleod 2020-05-21 07:52:30 +0000 May be related to #13604 87558 2 richard.purdie 2020-06-23 14:58:24 +0000 We saw this here: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1055 but I think I now understand how we might be able to reproduce it. I think this only fails when an oe-selftest fails. In this build on the autobuilder you will note this test failed in other builds: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1059 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1055 so i think if we have a failing test which we execute in a loop, we'd probably see this SystemExit issue occasionally. That suggests its something to do with races in the exit path when failures occur. All this is a guess but probably worth exploring. 87634 3 richard.purdie 2020-06-27 01:01:58 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1077/steps/8/logs/step2d 87729 4 randy.macleod 2020-07-09 08:09:43 +0000 May be related to 13978 according to Richard Purdie. Any news Kia? 87751 5 richard.purdie 2020-07-13 04:49:41 +0000 I think/hope this may be related to http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ab8ebfd35a7bbfec3dad612c824b29ca93dbe196 87830 6 richard.purdie 2020-07-23 08:33:45 +0000 We've not seen this issue, resolving with the above change, will reopen if it recurs.