2019-11-24 01:14:36,890 - oe-selftest - INFO - reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase) 2019-11-24 01:14:36,891 - oe-selftest - INFO - ... ERROR 2019-11-24 01:14:36,891 - oe-selftest - INFO - 12: 8/9 368/378 (4549.05s) (reproducible.ReproducibleTests.test_reproducible_builds) 2019-11-24 01:14:36,891 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds' 2019-11-24 01:17:10,587 - oe-selftest - INFO - ====================================================================== 2019-11-24 01:17:10,588 - oe-selftest - INFO - ERROR: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase) 2019-11-24 01:17:10,588 - oe-selftest - INFO - ---------------------------------------------------------------------- 2019-11-24 01:17:10,588 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds'
Targeting this bug specifically at the "lost connection" error. Its not test specific, you can do: diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py index 78189a4e6d..61138f608a 100644 --- a/meta/lib/oeqa/selftest/cases/reproducible.py +++ b/meta/lib/oeqa/selftest/cases/reproducible.py @@ -128,6 +128,8 @@ class ReproducibleTests(OESelftestTestCase): shutil.copyfile(source, dest) def test_reproducible_builds(self): + self.fail("The following %s packages are missing or different") + capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes] pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds 2019-11-27 21:31:14,859 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-11-27 21:31:20,750 - oe-selftest - INFO - Adding layer libraries: 2019-11-27 21:31:20,751 - oe-selftest - INFO - /home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib 2019-11-27 21:31:20,751 - oe-selftest - INFO - /home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib 2019-11-27 21:31:20,751 - oe-selftest - INFO - /home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib 2019-11-27 21:31:20,754 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-11-27 21:31:22,758 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build/conf/local.conf 2019-11-27 21:31:22,759 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-11-27 21:31:22,760 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests) 2019-11-27 21:31:24,898 - oe-selftest - INFO - ... FAIL 2019-11-27 21:31:24,898 - oe-selftest - INFO - Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 131, in test_reproducible_builds self.fail("The following %s packages are missing or different") AssertionError: The following %s packages are missing or different 2019-11-27 21:31:24,899 - oe-selftest - INFO - ====================================================================== 2019-11-27 21:31:24,900 - oe-selftest - INFO - FAIL: test_reproducible_builds (reproducible.ReproducibleTests) 2019-11-27 21:31:24,900 - oe-selftest - INFO - ---------------------------------------------------------------------- 2019-11-27 21:31:24,900 - oe-selftest - INFO - Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 131, in test_reproducible_builds self.fail("The following %s packages are missing or different") AssertionError: The following %s packages are missing or different 2019-11-27 21:31:24,900 - oe-selftest - INFO - ---------------------------------------------------------------------- 2019-11-27 21:31:24,900 - oe-selftest - INFO - Ran 1 test in 2.142s 2019-11-27 21:31:24,901 - oe-selftest - INFO - FAILED 2019-11-27 21:31:24,901 - oe-selftest - INFO - (failures=1) 2019-11-27 21:31:30,581 - oe-selftest - INFO - RESULTS: 2019-11-27 21:31:30,581 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2.14s) 2019-11-27 21:31:30,590 - oe-selftest - INFO - SUMMARY: 2019-11-27 21:31:30,590 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2.144s 2019-11-27 21:31:30,591 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=1, errors=0) pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds -j 2 2019-11-27 21:31:49,404 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-11-27 21:31:55,365 - oe-selftest - INFO - Adding layer libraries: 2019-11-27 21:31:55,366 - oe-selftest - INFO - /home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib 2019-11-27 21:31:55,366 - oe-selftest - INFO - /home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib 2019-11-27 21:31:55,367 - oe-selftest - INFO - /home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib 2019-11-27 21:31:55,369 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-11-27 21:32:00,153 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-9396/conf/local.conf 2019-11-27 21:32:00,154 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-11-27 21:32:03,351 - oe-selftest - INFO - reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase) 2019-11-27 21:32:03,352 - oe-selftest - INFO - ... ERROR 2019-11-27 21:32:03,353 - oe-selftest - INFO - 0: 1/1 1/1 (3.20s) (reproducible.ReproducibleTests.test_reproducible_builds) 2019-11-27 21:32:03,353 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds' 2019-11-27 21:32:03,353 - oe-selftest - INFO - ====================================================================== 2019-11-27 21:32:03,354 - oe-selftest - INFO - ERROR: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase) 2019-11-27 21:32:03,354 - oe-selftest - INFO - ---------------------------------------------------------------------- 2019-11-27 21:32:03,354 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds' 2019-11-27 21:32:03,354 - oe-selftest - INFO - ---------------------------------------------------------------------- 2019-11-27 21:32:03,355 - oe-selftest - INFO - Ran 1 test in 5.496s 2019-11-27 21:32:03,355 - oe-selftest - INFO - FAILED 2019-11-27 21:32:03,355 - oe-selftest - INFO - (errors=1) 2019-11-27 21:32:09,174 - oe-selftest - INFO - RESULTS: 2019-11-27 21:32:09,175 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: ERROR (3.20s) 2019-11-27 21:32:09,184 - oe-selftest - INFO - SUMMARY: 2019-11-27 21:32:09,185 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 5.901s 2019-11-27 21:32:09,185 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1) i.e. it only occurs with -j X and it only seems to happen on some hosts, here we see it on dbian8. There are probably other hosts it occurs on too like centos. It only occurs when a failure in a test occurs.
Managed to get a traceback which should help a lot: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/utils/concurrencytest.py", line 144, in _addResult foo = method(test, *args, **kwargs) File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/test_results.py", line 143, in addFailure return self.super.addFailure(test, err, details=details) File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/test_results.py", line 69, in addFailure return self.decorated.addFailure(test, err, details=details) File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/testtools/testresult/real.py", line 1404, in addFailure return self.decorated.addFailure(test, err) File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/__init__.py", line 677, in addFailure self._addOutcome("failure", test, error=error, details=details) File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/__init__.py", line 708, in _addOutcome tb_content = TracebackContent(error, test) File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/testtools/content.py", line 204, in __init__ stack_lines = list(traceback.TracebackException(exctype, value, tb, AttributeError: 'NoneType' object has no attribute 'TracebackException'
That worker uses our buildtools tarball which does not include python-traceback2 which subunit uses.
We can just use traceback so we should patch testtools to do that.
Patch on the mailing list.
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=19e3031e3a1e600e417625997fad11f708a1b3b5