<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>13652</bug_id>
          
          <creation_ts>2019-11-24 04:39:16 +0000</creation_ts>
          <short_desc>oe-selftest testtools.testresult.real._StringException: lost connection during failure</short_desc>
          <delta_ts>2019-11-28 21:46:31 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>OE-Core</product>
          <component>core</component>
          <version>3.1</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc>https://errors.yoctoproject.org/Errors/Details/282873/</bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>High</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>3.1</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Armin Kuster">akuster</reporter>
          <assigned_to name="Richard Purdie">richard.purdie</assigned_to>
          <cc>meta.mr.watcher</cc>
    
    <cc>meta.watcher</cc>
    
    <cc>richard.purdie</cc>
    
    <cc>tvgamblin</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Don&apos;t know</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>85623</commentid>
    <comment_count>0</comment_count>
    <who name="Armin Kuster">akuster</who>
    <bug_when>2019-11-24 04:39:16 +0000</bug_when>
    <thetext>2019-11-24 01:14:36,890 - oe-selftest - INFO - reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-24 01:14:36,891 - oe-selftest - INFO -  ... ERROR
2019-11-24 01:14:36,891 - oe-selftest - INFO - 12: 8/9 368/378 (4549.05s) (reproducible.ReproducibleTests.test_reproducible_builds)
2019-11-24 01:14:36,891 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test &apos;reproducible.ReproducibleTests.test_reproducible_builds&apos;

2019-11-24 01:17:10,587 - oe-selftest - INFO - ======================================================================
2019-11-24 01:17:10,588 - oe-selftest - INFO - ERROR: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-24 01:17:10,588 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-24 01:17:10,588 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test &apos;reproducible.ReproducibleTests.test_reproducible_builds&apos;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85647</commentid>
    <comment_count>1</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-11-27 21:37:53 +0000</bug_when>
    <thetext>Targeting this bug specifically at the &quot;lost connection&quot; error. Its not test specific, you can do:

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 78189a4e6d..61138f608a 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -128,6 +128,8 @@ class ReproducibleTests(OESelftestTestCase):
         shutil.copyfile(source, dest)
 
     def test_reproducible_builds(self):
+        self.fail(&quot;The following %s packages are missing or different&quot;)
+
         capture_vars = [&apos;DEPLOY_DIR_&apos; + c.upper() for c in self.package_classes]



pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
2019-11-27 21:31:14,859 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-11-27 21:31:20,750 - oe-selftest - INFO - Adding layer libraries:
2019-11-27 21:31:20,751 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib
2019-11-27 21:31:20,751 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib
2019-11-27 21:31:20,751 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib
2019-11-27 21:31:20,754 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-11-27 21:31:22,758 - oe-selftest - INFO - Adding: &quot;include selftest.inc&quot; in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build/conf/local.conf
2019-11-27 21:31:22,759 - oe-selftest - INFO - Adding: &quot;include bblayers.inc&quot; in bblayers.conf
2019-11-27 21:31:22,760 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests)
2019-11-27 21:31:24,898 - oe-selftest - INFO -  ... FAIL
2019-11-27 21:31:24,898 - oe-selftest - INFO - Traceback (most recent call last):
  File &quot;/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/reproducible.py&quot;, line 131, in test_reproducible_builds
    self.fail(&quot;The following %s packages are missing or different&quot;)
AssertionError: The following %s packages are missing or different

2019-11-27 21:31:24,899 - oe-selftest - INFO - ======================================================================
2019-11-27 21:31:24,900 - oe-selftest - INFO - FAIL: test_reproducible_builds (reproducible.ReproducibleTests)
2019-11-27 21:31:24,900 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:31:24,900 - oe-selftest - INFO - Traceback (most recent call last):
  File &quot;/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/reproducible.py&quot;, line 131, in test_reproducible_builds
    self.fail(&quot;The following %s packages are missing or different&quot;)
AssertionError: The following %s packages are missing or different

2019-11-27 21:31:24,900 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:31:24,900 - oe-selftest - INFO - Ran 1 test in 2.142s
2019-11-27 21:31:24,901 - oe-selftest - INFO - FAILED
2019-11-27 21:31:24,901 - oe-selftest - INFO -  (failures=1)
2019-11-27 21:31:30,581 - oe-selftest - INFO - RESULTS:
2019-11-27 21:31:30,581 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2.14s)
2019-11-27 21:31:30,590 - oe-selftest - INFO - SUMMARY:
2019-11-27 21:31:30,590 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2.144s
2019-11-27 21:31:30,591 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=1, errors=0)
pokybuild@debian8-ty-1:~/yocto-worker/oe-selftest-debian/build/build$ oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds -j 2
2019-11-27 21:31:49,404 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-11-27 21:31:55,365 - oe-selftest - INFO - Adding layer libraries:
2019-11-27 21:31:55,366 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib
2019-11-27 21:31:55,366 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-yocto-bsp/lib
2019-11-27 21:31:55,367 - oe-selftest - INFO - 	/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta-selftest/lib
2019-11-27 21:31:55,369 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-11-27 21:32:00,153 - oe-selftest - INFO - Adding: &quot;include selftest.inc&quot; in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-9396/conf/local.conf
2019-11-27 21:32:00,154 - oe-selftest - INFO - Adding: &quot;include bblayers.inc&quot; in bblayers.conf
2019-11-27 21:32:03,351 - oe-selftest - INFO - reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-27 21:32:03,352 - oe-selftest - INFO -  ... ERROR
2019-11-27 21:32:03,353 - oe-selftest - INFO - 0: 1/1 1/1 (3.20s) (reproducible.ReproducibleTests.test_reproducible_builds)
2019-11-27 21:32:03,353 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test &apos;reproducible.ReproducibleTests.test_reproducible_builds&apos;

2019-11-27 21:32:03,353 - oe-selftest - INFO - ======================================================================
2019-11-27 21:32:03,354 - oe-selftest - INFO - ERROR: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
2019-11-27 21:32:03,354 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:32:03,354 - oe-selftest - INFO - testtools.testresult.real._StringException: lost connection during failure report of test &apos;reproducible.ReproducibleTests.test_reproducible_builds&apos;

2019-11-27 21:32:03,354 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-11-27 21:32:03,355 - oe-selftest - INFO - Ran 1 test in 5.496s
2019-11-27 21:32:03,355 - oe-selftest - INFO - FAILED
2019-11-27 21:32:03,355 - oe-selftest - INFO -  (errors=1)
2019-11-27 21:32:09,174 - oe-selftest - INFO - RESULTS:
2019-11-27 21:32:09,175 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: ERROR (3.20s)
2019-11-27 21:32:09,184 - oe-selftest - INFO - SUMMARY:
2019-11-27 21:32:09,185 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 5.901s
2019-11-27 21:32:09,185 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)

i.e. it only occurs with -j X and it only seems to happen on some hosts, here we see it on dbian8. There are probably other hosts it occurs on too like centos. It only occurs when a failure in a test occurs.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85650</commentid>
    <comment_count>2</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-11-27 23:21:39 +0000</bug_when>
    <thetext>Managed to get a traceback which should help a lot:

Traceback (most recent call last):
  File &quot;/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/utils/concurrencytest.py&quot;, line 144, in _addResult
    foo = method(test, *args, **kwargs)
  File &quot;/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/test_results.py&quot;, line 143, in addFailure
    return self.super.addFailure(test, err, details=details)
  File &quot;/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/test_results.py&quot;, line 69, in addFailure
    return self.decorated.addFailure(test, err, details=details)
  File &quot;/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/testtools/testresult/real.py&quot;, line 1404, in addFailure
    return self.decorated.addFailure(test, err)
  File &quot;/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/__init__.py&quot;, line 677, in addFailure
    self._addOutcome(&quot;failure&quot;, test, error=error, details=details)
  File &quot;/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/subunit/__init__.py&quot;, line 708, in _addOutcome
    tb_content = TracebackContent(error, test)
  File &quot;/opt/poky/3.0/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/site-packages/testtools/content.py&quot;, line 204, in __init__
    stack_lines = list(traceback.TracebackException(exctype, value, tb,
AttributeError: &apos;NoneType&apos; object has no attribute &apos;TracebackException&apos;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85651</commentid>
    <comment_count>3</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-11-27 23:44:44 +0000</bug_when>
    <thetext>That worker uses our buildtools tarball which does not include python-traceback2 which subunit uses.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85652</commentid>
    <comment_count>4</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-11-27 23:48:10 +0000</bug_when>
    <thetext>We can just use traceback so we should patch testtools to do that.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85657</commentid>
    <comment_count>5</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-11-28 12:41:44 +0000</bug_when>
    <thetext>Patch on the mailing list.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85666</commentid>
    <comment_count>6</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-11-28 21:46:31 +0000</bug_when>
    <thetext>http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=19e3031e3a1e600e417625997fad11f708a1b3b5</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>