Created attachment 4717 [details] corresponding cooker log showing a process left behind. Bitbake server holding off shutdown due to left over processes: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1280 2020-08-27 10:09:54,351 - oe-selftest - INFO - 10: 25/27 128/413 (171.36s) (recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfiles_basic) 2020-08-27 10:09:54,351 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 715, in test_recipetool_appendsrcfiles_basic self._test_appendsrcfiles('gcc', newfiles, destdir=destdir, options='-W') File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 634, in _test_appendsrcfiles self._try_recipetool_appendsrcfiles(testrecipe, newfiles, expectedfiles=expectedfiles, destdir=destdir, options=options) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 575, in _try_recipetool_appendsrcfiles return self._try_recipetool_appendcmd(cmd, testrecipe, expectedfiles, expectedlines) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 49, in _try_recipetool_appendcmd bbappendfile = self._check_bbappend(testrecipe, recipefile, self.templayerdir) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 186, in _check_bbappend result = runCmd('bitbake-layers show-appends', cwd=self.builddir) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) AssertionError: Command 'bitbake-layers show-appends' returned non-zero exit status 1: NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#1)... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#2)... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#3)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#3)... NOTE: Retrying server connection (#3)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#4)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#4)... NOTE: Retrying server connection (#4)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#5)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#5)... NOTE: Retrying server connection (#5)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#6)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#6)... NOTE: Retrying server connection (#6)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#7)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#7)... NOTE: Retrying server connection (#7)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#8)... ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log). NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#8)... ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log). 2020-08-27 10:12:41,030 - oe-selftest - INFO - recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfiles_basic_subdir (subunit.RemotedTestCase) 2020-08-27 10:12:41,031 - oe-selftest - INFO - ... FAIL 2020-08-27 10:12:41,031 - oe-selftest - INFO - 10: 26/27 129/413 (166.73s) (recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfiles_basic_subdir) 2020-08-27 10:12:41,031 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 718, in test_recipetool_appendsrcfiles_basic_subdir self.test_recipetool_appendsrcfiles_basic(destdir='testdir') File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 715, in test_recipetool_appendsrcfiles_basic self._test_appendsrcfiles('gcc', newfiles, destdir=destdir, options='-W') File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 634, in _test_appendsrcfiles self._try_recipetool_appendsrcfiles(testrecipe, newfiles, expectedfiles=expectedfiles, destdir=destdir, options=options) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 575, in _try_recipetool_appendsrcfiles return self._try_recipetool_appendcmd(cmd, testrecipe, expectedfiles, expectedlines) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 49, in _try_recipetool_appendcmd bbappendfile = self._check_bbappend(testrecipe, recipefile, self.templayerdir) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 186, in _check_bbappend result = runCmd('bitbake-layers show-appends', cwd=self.builddir) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) AssertionError: Command 'bitbake-layers show-appends' returned non-zero exit status 1: NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#1)... NOTE: Retrying server connection (#1)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#2)... NOTE: Retrying server connection (#2)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#3)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#3)... NOTE: Retrying server connection (#3)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#4)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#4)... NOTE: Retrying server connection (#4)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#5)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#5)... NOTE: Retrying server connection (#5)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#6)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#6)... NOTE: Retrying server connection (#6)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#7)... NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#7)... NOTE: Retrying server connection (#7)... NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#8)... ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log). NOTE: Reconnecting to bitbake server... NOTE: Previous bitbake instance shutting down?, waiting to retry... NOTE: Retrying server connection (#8)... ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log). 2020-08-27 10:17:15,274 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase) 2020-08-27 10:17:15,274 - oe-selftest - INFO - ... ok Attached the corresponding cooker log showing a process left behind.
Likely fixed by changing the write-back mode for the filesystem on the builder to journal file metadata but not contents.
As Randy mentions, we changed the ext4 filesystems to use writeback journaling. We think the cause was IO priorities being lost due to the journal running in a single process, when combined with heavy IO from say a kernel build and backlogged janitor file deletion.