Bug 14032 - bitbake timeouts in oe-selftest
Summary: bitbake timeouts in oe-selftest
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.2 M4
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard: AB-INT
Depends on:
Blocks:
 
Reported: 2020-09-01 00:27 UTC by Alexander Kanavin
Modified: 2020-10-21 02:55 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kanavin 2020-09-01 00:27:00 UTC
Seen here:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1290/steps/8/logs/step2d

  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 322, in test_postinst_rootfs_and_boot_systemd
    self.init_manager_loop("systemd")
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 273, in init_manager_loop
    with runqemu('core-image-minimal') as qemu:
  File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 316, in runqemu
    tinfoil.prepare(config_only=False, quiet=True)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tinfoil.py", line 412, in prepare
    self.run_actions(config_params)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tinfoil.py", line 425, in run_actions
    ret = self.ui_module.main(self.server_connection.connection, self.server_connection.events, config_params)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/ui/knotty.py", line 386, in main
    params.updateToServer(server, os.environ.copy())
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer
    ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 367, in runCommand
    raise ProcessTimeout("Timeout while waiting for a reply from the bitbake server (60s)")
bb.server.process.ProcessTimeout: Timeout while waiting for a reply from the bitbake server (60s)


  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 300, in test_postinst_rootfs_and_boot_sysvinit
    self.init_manager_loop("sysvinit")
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 247, in init_manager_loop
    vars = get_bb_vars(("IMAGE_ROOTFS", "sysconfdir"), "core-image-minimal")
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 237, in get_bb_vars
    bbenv = get_bb_env(target, postconfig=postconfig)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 231, in get_bb_env
    return bitbake("-e %s" % target, postconfig=postconfig).output
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 223, in bitbake
    return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake  -e core-image-minimal' returned non-zero exit status 1:
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#3)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#4)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#5)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#6)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#7)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#8)...
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).
Comment 1 Richard Purdie 2020-10-08 00:07:33 UTC
Still happening https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1424/steps/8/logs/step2d even if much more rarely
Comment 2 Richard Purdie 2020-10-19 05:53:19 UTC
Still happening:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1469/steps/8/logs/step2d

Propose we move the sync call to command execution.
Comment 3 Richard Purdie 2020-10-19 05:57:33 UTC
Also:

devtool.DevtoolUpdateTests.test_devtool_update_recipe_append_git on debian9-ty-2

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1453/steps/8/logs/step2d
Comment 4 Richard Purdie 2020-10-19 05:58:54 UTC
Also debian9-ty-2 signing.Signing.test_signing_packages 
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1453/steps/8/logs/step2d
Comment 5 Richard Purdie 2020-10-21 02:55:25 UTC
After lots of attempts, this is probably the best option we have:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=dfc54a477bca7773d075041156897867cfc8ad67