Bug 13652 - oe-selftest testtools.testresult.real._StringException: lost connection during failure
Summary: oe-selftest testtools.testresult.real._StringException: lost connection durin...
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.1
Hardware: x86 Multiple
: High normal
Target Milestone: 3.1
Assignee: Richard Purdie
QA Contact:
URL: https://errors.yoctoproject.org/Error...
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-11-24 04:39 UTC by Armin Kuster
Modified: 2019-11-28 21:46 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Kuster 2019-11-24 04:39:16 UTC
2019-11-24 01:14:36,890 - oe-selftest - INFO - reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-24 01:14:36,891 - oe-selftest - INFO -  ... ERROR
2019-11-24 01:14:36,891 - oe-selftest - INFO - 12: 8/9 368/378 (4549.05s) (reproducible.ReproducibleTests.test_reproducible_builds)
2019-11-24 01:14:36,891 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds'

2019-11-24 01:17:10,587 - oe-selftest - INFO - ======================================================================
2019-11-24 01:17:10,588 - oe-selftest - INFO - ERROR: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-24 01:17:10,588 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-24 01:17:10,588 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds'
Comment 1 Richard Purdie 2019-11-27 21:37:53 UTC
Targeting this bug specifically at the "lost connection" error. Its not test specific, you can do:

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 78189a4e6d..61138f608a 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -128,6 +128,8 @@ class ReproducibleTests(OESelftestTestCase):
         shutil.copyfile(source, dest)
 
     def test_reproducible_builds(self):
+        self.fail("The following %s packages are missing or different")
+
         capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes]



pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
2019-11-27 21:31:14,859 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-11-27 21:31:20,750 - oe-selftest - INFO - Adding layer libraries:
2019-11-27 21:31:20,751 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib
2019-11-27 21:31:20,751 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib
2019-11-27 21:31:20,751 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib
2019-11-27 21:31:20,754 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-11-27 21:31:22,758 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build/conf/local.conf
2019-11-27 21:31:22,759 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-11-27 21:31:22,760 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests)
2019-11-27 21:31:24,898 - oe-selftest - INFO -  ... FAIL
2019-11-27 21:31:24,898 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 131, in test_reproducible_builds
    self.fail("The following %s packages are missing or different")
AssertionError: The following %s packages are missing or different

2019-11-27 21:31:24,899 - oe-selftest - INFO - ======================================================================
2019-11-27 21:31:24,900 - oe-selftest - INFO - FAIL: test_reproducible_builds (reproducible.ReproducibleTests)
2019-11-27 21:31:24,900 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:31:24,900 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 131, in test_reproducible_builds
    self.fail("The following %s packages are missing or different")
AssertionError: The following %s packages are missing or different

2019-11-27 21:31:24,900 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:31:24,900 - oe-selftest - INFO - Ran 1 test in 2.142s
2019-11-27 21:31:24,901 - oe-selftest - INFO - FAILED
2019-11-27 21:31:24,901 - oe-selftest - INFO -  (failures=1)
2019-11-27 21:31:30,581 - oe-selftest - INFO - RESULTS:
2019-11-27 21:31:30,581 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2.14s)
2019-11-27 21:31:30,590 - oe-selftest - INFO - SUMMARY:
2019-11-27 21:31:30,590 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2.144s
2019-11-27 21:31:30,591 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=1, errors=0)
pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds -j 2
2019-11-27 21:31:49,404 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-11-27 21:31:55,365 - oe-selftest - INFO - Adding layer libraries:
2019-11-27 21:31:55,366 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib
2019-11-27 21:31:55,366 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib
2019-11-27 21:31:55,367 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib
2019-11-27 21:31:55,369 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-11-27 21:32:00,153 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-9396/conf/local.conf
2019-11-27 21:32:00,154 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-11-27 21:32:03,351 - oe-selftest - INFO - reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-27 21:32:03,352 - oe-selftest - INFO -  ... ERROR
2019-11-27 21:32:03,353 - oe-selftest - INFO - 0: 1/1 1/1 (3.20s) (reproducible.ReproducibleTests.test_reproducible_builds)
2019-11-27 21:32:03,353 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds'

2019-11-27 21:32:03,353 - oe-selftest - INFO - ======================================================================
2019-11-27 21:32:03,354 - oe-selftest - INFO - ERROR: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-27 21:32:03,354 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:32:03,354 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test 'reproducible.ReproducibleTests.test_reproducible_builds'

2019-11-27 21:32:03,354 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:32:03,355 - oe-selftest - INFO - Ran 1 test in 5.496s
2019-11-27 21:32:03,355 - oe-selftest - INFO - FAILED
2019-11-27 21:32:03,355 - oe-selftest - INFO -  (errors=1)
2019-11-27 21:32:09,174 - oe-selftest - INFO - RESULTS:
2019-11-27 21:32:09,175 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: ERROR (3.20s)
2019-11-27 21:32:09,184 - oe-selftest - INFO - SUMMARY:
2019-11-27 21:32:09,185 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 5.901s
2019-11-27 21:32:09,185 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)

i.e. it only occurs with -j X and it only seems to happen on some hosts, here we see it on dbian8. There are probably other hosts it occurs on too like centos. It only occurs when a failure in a test occurs.
Comment 2 Richard Purdie 2019-11-27 23:21:39 UTC
Managed to get a traceback which should help a lot:

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/utils/concurrencytest.py", line 144, in _addResult
    foo = method(test, *args, **kwargs)
  File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/test_results.py", line 143, in addFailure
    return self.super.addFailure(test, err, details=details)
  File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/test_results.py", line 69, in addFailure
    return self.decorated.addFailure(test, err, details=details)
  File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/testtools/testresult/real.py", line 1404, in addFailure
    return self.decorated.addFailure(test, err)
  File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/__init__.py", line 677, in addFailure
    self._addOutcome("failure", test, error=error, details=details)
  File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/__init__.py", line 708, in _addOutcome
    tb_content = TracebackContent(error, test)
  File "/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/testtools/content.py", line 204, in __init__
    stack_lines = list(traceback.TracebackException(exctype, value, tb,
AttributeError: 'NoneType' object has no attribute 'TracebackException'
Comment 3 Richard Purdie 2019-11-27 23:44:44 UTC
That worker uses our buildtools tarball which does not include python-traceback2 which subunit uses.
Comment 4 Richard Purdie 2019-11-27 23:48:10 UTC
We can just use traceback so we should patch testtools to do that.
Comment 5 Richard Purdie 2019-11-28 12:41:44 UTC
Patch on the mailing list.