Bug 14032

Summary:	bitbake timeouts in oe-selftest
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Alexander Kanavin <alex.kanavin>
Component:	bitbake	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	poky.bs.watcher, poky.watcher, randy.macleod
Version:	unspecified
Target Milestone:	3.2 M4
Hardware:	x86
OS:	Multiple
Whiteboard:	AB-INT
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Alexander Kanavin 2020-09-01 00:27:00 UTC

Seen here:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1290/steps/8/logs/step2d

  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 322, in test_postinst_rootfs_and_boot_systemd
    self.init_manager_loop("systemd")
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 273, in init_manager_loop
    with runqemu('core-image-minimal') as qemu:
  File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 316, in runqemu
    tinfoil.prepare(config_only=False, quiet=True)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tinfoil.py", line 412, in prepare
    self.run_actions(config_params)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tinfoil.py", line 425, in run_actions
    ret = self.ui_module.main(self.server_connection.connection, self.server_connection.events, config_params)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/ui/knotty.py", line 386, in main
    params.updateToServer(server, os.environ.copy())
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cookerdata.py", line 73, in updateToServer
    ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 367, in runCommand
    raise ProcessTimeout("Timeout while waiting for a reply from the bitbake server (60s)")
bb.server.process.ProcessTimeout: Timeout while waiting for a reply from the bitbake server (60s)


  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 300, in test_postinst_rootfs_and_boot_sysvinit
    self.init_manager_loop("sysvinit")
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 247, in init_manager_loop
    vars = get_bb_vars(("IMAGE_ROOTFS", "sysconfdir"), "core-image-minimal")
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 237, in get_bb_vars
    bbenv = get_bb_env(target, postconfig=postconfig)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 231, in get_bb_env
    return bitbake("-e %s" % target, postconfig=postconfig).output
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 223, in bitbake
    return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake  -e core-image-minimal' returned non-zero exit status 1:
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#3)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#4)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#5)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#6)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#7)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#8)...
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).

Comment 1 Richard Purdie 2020-10-08 00:07:33 UTC

Still happening https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1424/steps/8/logs/step2d even if much more rarely

Comment 2 Richard Purdie 2020-10-19 05:53:19 UTC

Still happening:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1469/steps/8/logs/step2d

Propose we move the sync call to command execution.

Comment 3 Richard Purdie 2020-10-19 05:57:33 UTC

Also:

devtool.DevtoolUpdateTests.test_devtool_update_recipe_append_git on debian9-ty-2

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1453/steps/8/logs/step2d

Comment 4 Richard Purdie 2020-10-19 05:58:54 UTC

Also debian9-ty-2 signing.Signing.test_signing_packages 
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1453/steps/8/logs/step2d

Comment 5 Richard Purdie 2020-10-21 02:55:25 UTC

After lots of attempts, this is probably the best option we have:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=dfc54a477bca7773d075041156897867cfc8ad67