Bug 13604 - [master-next] Distrodata.test_maintainers fails
Summary: [master-next] Distrodata.test_maintainers fails
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.0
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.2 M2
Assignee: Kai Kang
QA Contact:
URL: https://errors.yoctoproject.org/Error...
Whiteboard: AB-INT
Depends on:
Blocks:
 
Reported: 2019-10-20 15:51 UTC by Armin Kuster
Modified: 2020-07-23 08:33 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Kuster 2019-10-20 15:51:20 UTC
2019-10-19 10:28:12,096 - oe-selftest - INFO -  ... ERROR
2019-10-19 10:28:12,352 - oe-selftest - INFO - 2: 29/35 185/366 (178.56s) (distrodata.Distrodata.test_maintainers)
2019-10-19 10:28:12,352 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/distrodata.py", line 61, in test_maintainers
    tinfoil.prepare(config_only=False)
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/tinfoil.py", line 387, in prepare
    extrafeatures)
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/main.py", line 438, in setup_bitbake
    server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset)
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/server/process.py", line 447, in __init__
    raise SystemExit(1)
SystemExit: 1
Comment 1 Richard Purdie 2019-11-08 22:58:10 UTC
I talked a bit with Paul about this, it appears that something is going wrong with tinfoil's early exception handling when it can't setup/connect to a server. We likely need to fix the logging which would then let us get to the real underlying failure.
Comment 2 Richard Purdie 2019-12-08 10:40:33 UTC
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/538

2019-12-07 23:57:13,529 - oe-selftest - INFO - 2: 29/40 145/384 (138.08s) (distrodata.Distrodata.test_maintainers)
2019-12-07 23:57:13,529 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/distrodata.py", line 61, in test_maintainers
    tinfoil.prepare(config_only=False)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/tinfoil.py", line 387, in prepare
    extrafeatures)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/main.py", line 438, in setup_bitbake
    server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 447, in __init__
    raise SystemExit(1)
SystemExit: 1


2019-12-07 23:57:13,530 - oe-selftest - INFO - runqemu.RunqemuTests.test_boot_deploy (subunit.RemotedTestCase)
2019-12-07 23:57:13,549 - oe-selftest - INFO -  ... ERROR
2019-12-07 23:57:13,549 - oe-selftest - INFO - 0: 22/33 146/384 (1983.48s) (runqemu.RunqemuTests.test_boot_deploy)
2019-12-07 23:57:13,549 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/runqemu.py", line 93, in test_boot_deploy
    with runqemu(self.recipe, ssh=False, launch_cmd=cmd) as qemu:
  File "/opt/poky/3.0.1/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 314, in runqemu
    tinfoil.prepare(config_only=False, quiet=True)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/tinfoil.py", line 387, in prepare
    extrafeatures)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/main.py", line 438, in setup_bitbake
    server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 447, in __init__
    raise SystemExit(1)
SystemExit: 1
Comment 3 Richard Purdie 2020-01-16 15:54:55 UTC
I believe I spotted a pattern. This appears to only occur when a test is failing. I also wonder if there is a link to tests which "fork" since maintainers and reproducibility generate a lot of other processes.

Next steps may be to stress test with failing tests.
Comment 4 Randy MacLeod 2020-02-13 15:45:23 UTC
Kai, we'd like this done in the next week or two. Okay?
Comment 5 Richard Purdie 2020-02-20 16:12:45 UTC
I think the sys.exit(1) is related from the other bug and likely causing this.

*** This bug has been marked as a duplicate of bug 13664 ***
Comment 6 Richard Purdie 2020-02-24 09:16:27 UTC
Sadly https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/723 proves this was not caused by the other bug, reopening.
Comment 7 Richard Purdie 2020-03-05 23:11:22 UTC
Looks like this issue happens in different tests:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/751
Comment 8 Richard Purdie 2020-03-05 23:12:23 UTC
Pattern in the bug seems to always be on selftest-centos...
Comment 9 Randy MacLeod 2020-03-12 14:41:51 UTC
Kai is having trouble reproducing this bug. Try testing under high load. 
If anyone has another suggetion, please let us know. It's one of the most commonly occurring non-reproducible bugs that the autobuilder sees.
Comment 10 Randy MacLeod 2020-03-19 14:50:35 UTC
Not seen in the autobuilder for a few days but it will almost certainly come back. Add debugging and send a patch that can be tested in master-next, please.
Comment 11 Randy MacLeod 2020-04-02 14:41:19 UTC
Still no sign of this bug in the autobuilder but we'd like to get it fixed ASAP.
Comment 12 Richard Purdie 2020-07-13 04:49:05 UTC
I think/hope this may be related to http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ab8ebfd35a7bbfec3dad612c824b29ca93dbe196
Comment 13 Richard Purdie 2020-07-23 08:33:28 UTC
We've not seen this issue, resolving with the above change, will reopen if it recurs.