Bug 14027

Summary:

bitbake cooker processes left hanging at exit

Product:

[Build System, Metadata & Runtime] BitBake

Reporter:

Richard Purdie <richard.purdie>

Component:

bitbake

Assignee:

Richard Purdie <richard.purdie>

Status:

RESOLVED FIXED

QA Contact:

Severity:

normal

Priority:

High

CC:

poky.bs.watcher, poky.watcher, randy.macleod

Version:

3.2

Target Milestone:

3.2 M4

Hardware:

x86

OS:

Multiple

Whiteboard:

AB-INT

OS type for building Yocto:

---

Type of Regression:

---

Verified:

Documentation change:

No (bug/feature does not impact docs)

Attachments:

Description	Flags
corresponding cooker log showing a process left behind.	none

Description Richard Purdie 2020-08-27 13:38:28 UTC

Created attachment 4717 [details]
corresponding cooker log showing a process left behind.

Bitbake server holding off shutdown due to left over processes:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1280

2020-08-27 10:09:54,351 - oe-selftest - INFO - 10: 25/27 128/413 (171.36s) (recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfiles_basic)
2020-08-27 10:09:54,351 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 715, in test_recipetool_appendsrcfiles_basic
    self._test_appendsrcfiles('gcc', newfiles, destdir=destdir, options='-W')
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 634, in _test_appendsrcfiles
    self._try_recipetool_appendsrcfiles(testrecipe, newfiles, expectedfiles=expectedfiles, destdir=destdir, options=options)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 575, in _try_recipetool_appendsrcfiles
    return self._try_recipetool_appendcmd(cmd, testrecipe, expectedfiles, expectedlines)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 49, in _try_recipetool_appendcmd
    bbappendfile = self._check_bbappend(testrecipe, recipefile, self.templayerdir)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 186, in _check_bbappend
    result = runCmd('bitbake-layers show-appends', cwd=self.builddir)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake-layers show-appends' returned non-zero exit status 1:
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#1)...
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#2)...
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#3)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#3)...
NOTE: Retrying server connection (#3)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#4)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#4)...
NOTE: Retrying server connection (#4)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#5)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#5)...
NOTE: Retrying server connection (#5)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#6)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#6)...
NOTE: Retrying server connection (#6)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#7)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#7)...
NOTE: Retrying server connection (#7)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#8)...
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#8)...
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).


2020-08-27 10:12:41,030 - oe-selftest - INFO - recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfiles_basic_subdir (subunit.RemotedTestCase)
2020-08-27 10:12:41,031 - oe-selftest - INFO -  ... FAIL
2020-08-27 10:12:41,031 - oe-selftest - INFO - 10: 26/27 129/413 (166.73s) (recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfiles_basic_subdir)
2020-08-27 10:12:41,031 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 718, in test_recipetool_appendsrcfiles_basic_subdir
    self.test_recipetool_appendsrcfiles_basic(destdir='testdir')
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 715, in test_recipetool_appendsrcfiles_basic
    self._test_appendsrcfiles('gcc', newfiles, destdir=destdir, options='-W')
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 634, in _test_appendsrcfiles
    self._try_recipetool_appendsrcfiles(testrecipe, newfiles, expectedfiles=expectedfiles, destdir=destdir, options=options)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 575, in _try_recipetool_appendsrcfiles
    return self._try_recipetool_appendcmd(cmd, testrecipe, expectedfiles, expectedlines)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 49, in _try_recipetool_appendcmd
    bbappendfile = self._check_bbappend(testrecipe, recipefile, self.templayerdir)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 186, in _check_bbappend
    result = runCmd('bitbake-layers show-appends', cwd=self.builddir)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake-layers show-appends' returned non-zero exit status 1:
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#1)...
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#2)...
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#3)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#3)...
NOTE: Retrying server connection (#3)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#4)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#4)...
NOTE: Retrying server connection (#4)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#5)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#5)...
NOTE: Retrying server connection (#5)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#6)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#6)...
NOTE: Retrying server connection (#6)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#7)...
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#7)...
NOTE: Retrying server connection (#7)...
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#8)...
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).
NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection (#8)...
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).


2020-08-27 10:17:15,274 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase)
2020-08-27 10:17:15,274 - oe-selftest - INFO -  ... ok

Attached the corresponding cooker log showing a process left behind.

Comment 1 Randy MacLeod 2020-09-17 07:42:52 UTC

Likely fixed by changing the write-back mode for the filesystem on the builder to journal file metadata but not contents.

Comment 2 Richard Purdie 2020-09-24 04:34:46 UTC

As Randy mentions, we changed the ext4 filesystems to use writeback journaling. We think the cause was IO priorities being lost due to the journal running in a single process, when combined with heavy IO from say a kernel build and backlogged janitor file deletion.