Bug 11911

Summary: connection lost between host-target on a nightly-arm-lsb AB build
Product: [QA/Testing] Runtime Testing Reporter: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Component: generalAssignee: Kristi <kristi>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: joshuagloe, srifenbark
Version: 2.4   
Target Milestone: 2.5 M2   
Hardware: x86   
OS: Multiple   
Whiteboard: 19 DECEMBER, 2017: RESOLVED
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Done (doc changes complete)
Attachments:
Description Flags
AB log containg the error, taken from [1] none

Description Leonardo Sandoval Gonzalez 2017-08-08 21:07:22 UTC
Created attachment 3925 [details]
AB log containg the error, taken from [1]

For some reason, AB build [1], while doing do_testimage testing, connection between host and target was lost, thus failing several tests. Below is a single error log

[1] https://autobuilder.yocto.io/builders/nightly-arm-lsb/builds/398

--snippet--
NOTE: core-image-lsb-1.0-r0 do_testimage: ======================================================================
NOTE: core-image-lsb-1.0-r0 do_testimage: ERROR: setUpClass (rpm.RpmInstallRemoveTest)
NOTE: core-image-lsb-1.0-r0 do_testimage: ----------------------------------------------------------------------
NOTE: core-image-lsb-1.0-r0 do_testimage: Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/runtime/cases/rpm.py", line 47, in setUpClass
    cls.tc.target.copyTo(test_file, dst)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/core/target/ssh.py", line 103, in copyTo
    return self._run(scpCmd, ignore_status=False)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/core/target/ssh.py", line 61, in _run
    "status %d:\n%s" % (command, status, output))
AssertionError: Command '['scp', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/deploy/rpm/armv5e/rpm-doc-4.1\
3.90+git0+a8e51b3bb0-r0.0.armv5e.rpm', 'root@192.168.7.4:/tmp/rpm-doc.rpm']' returned non-zero exit status 1:
ssh: connect to host 192.168.7.4 port 22: No route to host^M
lost connection
--snippet--


Tests failing

    969:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_dnf_install_dependency (dnf.DnfRepoTest)
   1020:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_dnf_reinstall (dnf.DnfRepoTest)
   1037:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_dnf_repoinfo (dnf.DnfRepoTest)
   1054:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_syslog_running (oe_syslog.SyslogTest)
   1068:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_pam (pam.PamBasicTest)
   1082:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_rpm_help (rpm.RpmBasicTest)
   1094:NOTE: core-image-lsb-1.0-r0 do_testimage: FAIL: test_scp_file (scp.ScpTest)
   1448:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_connmand_help (connman.ConnmanTest)
   1462:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_date (date.DateTest)
   1474:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_dnf_help (dnf.DnfBasicTest)
   1492:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_ldd_exists (ldd.LddTest)
   1506:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_1_logrotate_setup (logrotate.LogrotateTest)
   1520:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_syslog_running (oe_syslog.SyslogTest)
   1534:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_pam (pam.PamBasicTest)
   1548:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_rpm_help (rpm.RpmBasicTest)
   1560:NOTE: core-image-lsb-sdk-1.0-r0 do_testimage: FAIL: test_scp_file (scp.ScpTest)
Comment 1 Joshua Lock 2017-08-10 15:15:55 UTC
We've seen this a couple of times on the tumbleweed builder on the new cluster, Michael is going to take a look.
Comment 2 Michael Halstead 2017-08-30 18:05:44 UTC
I uninstalled Wicked and installed NetworkManager and standard configuration for it last week. I'm seeing solid green at https://autobuilder.yocto.io/buildslaves/tumbleweed.yocto.io so I think that was an effective solution.

Where can we document that sanity testing and https://en.opensuse.org/Portal:Wicked don't work reliably together?
Comment 3 Leonardo Sandoval Gonzalez 2017-08-30 19:17:29 UTC
(In reply to comment #2)
> I uninstalled Wicked and installed NetworkManager and standard configuration
> for it last week. I'm seeing solid green at
> https://autobuilder.yocto.io/buildslaves/tumbleweed.yocto.io so I think that
> was an effective solution.
> 
> Where can we document that sanity testing and
> https://en.opensuse.org/Portal:Wicked don't work reliably together?

not really sure where but CC Jose, from QA team.
Comment 4 Joshua Lock 2017-09-15 12:25:20 UTC
Scott, would a comment on the incompatibility between the Wicked network manager and our sanity tests be appropriate in the System Requirements section of the Reference Manual?
Comment 5 Scott Rifenbark 2017-09-21 23:10:30 UTC
Yes - Here are some documentation fixes we can put in place:

http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#testing-and-quality-assurance - We can add a note here indicating sanity testing and https://en.opensuse.org/Portal:Wicked don't work reliably together.

http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#ref-classes-insane - We can add a sentence at the end of the second paragraph to reference the "Testing and Quality Assurance" section (http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#testing-and-quality-assurance) for information about QA testing of the Yocto Project.

We can alter the first sentence of the second paragraph in the http://www.yoctoproject.org/docs/2.4/dev-manual/dev-manual.html#performing-automated-runtime-testing section to say "For information and limitations on the test...".  This should suffice since we will have a new note in that section on "Testing and Quality Assurance", which is referenced. 

For the http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#required-packages-for-the-host-development-system section, we can add a note at each of the four distribution sub-sections right below the "OpenEmbedded Self-Test" bullet that says sanity testing and https://en.opensuse.org/Portal:Wicked don't work reliably together.

Assigning to Kristi.
Comment 6 Joshua Lock 2017-09-22 09:11:00 UTC
(In reply to comment #5)
> For the
> http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#required-
> packages-for-the-host-development-system section, we can add a note at each
> of the four distribution sub-sections right below the "OpenEmbedded
> Self-Test" bullet that says sanity testing and
> https://en.opensuse.org/Portal:Wicked don't work reliably together.

It would probably suffice to add that note only to the openSUSE section, I'm not aware of other distros shipping Wicked.
Comment 7 Scott Rifenbark 2017-09-22 13:29:08 UTC
Thanks Joshua... good to know.  We can qualify the note then also.

Scott
Comment 8 Kristi 2017-12-18 23:18:56 UTC
Josh - Based on your Comment 6 I've added the requested Note. Would you please review and let me know if changes are necessary at http://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#opensuse-packages

Thanks!
Comment 9 Joshua Lock 2017-12-19 12:10:10 UTC
Can we be a little less terse, something like:

"Sanity testing, via the testimage classes, does not work on systems using the Wicked network manager."

testimage should link to ref-classes-testimage*
Comment 10 Kristi 2017-12-19 21:45:10 UTC
Thank you Josh! I've updated the Note here: http://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#opensuse-packages and marking this to RESOLVED/Done