Bug 15327

Summary: AB-INT: ssh.SSHTest.test_ssh fails with "No route to host"
Product: [QA/Testing] Runtime Testing Reporter: Alexandre Belloni <alexandre.belloni>
Component: oeqa/runtimeAssignee: Unassigned <unassigned>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: alexandre.belloni, chee.yang.lee, mathieu.dubois-briand, randy.macleod, richard.purdie
Version: unspecified   
Target Milestone: 5.99   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Alexandre Belloni 2023-12-16 23:20:55 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/5857/steps/12/logs/stdio

This failed with a bunch:
RESULTS - ssh.SSHTest.test_ssh: FAILED (477.53s)
SUMMARY:
core-image-ptest-libtest-needs-perl () - Ran 6 tests in 477.562s
core-image-ptest-libtest-needs-perl - FAIL - Required tests failed (successes=2, skipped=3, failures=1, errors=0)
ERROR: core-image-ptest-libtest-needs-perl-1.0-r0 do_testimage: core-image-ptest-libtest-needs-perl - FAILED - also check the logs in /home/pokybuild/yocto-worker/qemuarm64-ptest/build/build/tmp/log
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/qemuarm64-ptest/build/build/tmp/work/qemuarm64-poky-linux/core-image-ptest-libtest-needs-perl/1.0/temp/log.do_testimage.3497123
NOTE: recipe core-image-ptest-libtest-needs-perl-1.0-r0: task do_testimage: Failed
ERROR: Task (virtual:mcextend:libtest-needs-perl:/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/recipes-core/images/core-image-ptest.bb:do_testimage) failed with exit code '1'
NOTE: Pressure status changed to CPU: True, IO: True, Mem: None (CPU: 14694.4/10000.0, IO: 892491.2/10000.0, Mem: 0.0/None) - using 8/16 bitbake threads
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm64-ptest/build/meta/lib/oeqa/runtime/cases/ssh.py", line 20, in test_ssh
    self.assertEqual(status, 0, msg='SSH Test failed: %s' % output)
AssertionError: 255 != 0 : SSH Test failed: ssh: connect to host 192.168.7.2 port 22: No route to host
Comment 1 Alexandre Belloni 2023-12-16 23:23:40 UTC 
RP investigated and said:

10:13:53             RP | abelloni: can you see a timestamp for when it happened
10:13:54             RP | ?
10:14:35       abelloni | 1702627323
10:14:52       abelloni | $ LC_TIME=C date -u -d @1702627323
10:14:55       abelloni | Fri Dec 15 08:02:03 UTC 2023
10:15:29       abelloni | Thu Dec 14 23:11:30 UTC 2023
10:15:37       abelloni | between those times
10:16:33             RP | at 6:24 there is a systemd-networkd-wait-online: Timeout occurred waiting for network connectivity
10:18:50             RP | was part of an apt update. Nothing else I can see
10:22:13             RP | abelloni: I think networkd probably did something :/
Comment 2 Alexandre Belloni 2023-12-16 23:32:00 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/93/builds/1831/steps/12/logs/stdio

qemuarm64-ptest-fast ubuntu2204-arm-1
Comment 3 Alexandre Belloni 2023-12-16 23:33:58 UTC 
This one was running between:

Fri Dec 15 01:07:14 UTC 2023
Fri Dec 15 03:12:39 UTC 2023
Comment 4 Chee Yang 2023-12-18 06:29:25 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/5869/steps/12/logs/stdio
ubuntu2204-arm-1  qemuarm64-ptest
Comment 5 Alexandre Belloni 2023-12-18 17:07:03 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/5869/steps/12/logs/stdio

qemuarm64-ptest ubuntu2204-arm-1
Comment 6 Randy MacLeod 2023-12-21 15:42:19 UTC 
Just looks like the networking went dead. Systemd-networkd notes a change in carrier but it's not clear if our log message happens before that log.
Comment 7 Randy MacLeod 2024-01-25 14:28:47 UTC 
Bulk move of bugs owned by "unassigned" from 5.0-M2 to 5.0-M3.
Comment 8 Randy MacLeod 2024-02-22 19:53:23 UTC 
Bulk move of bugs from 5.0-M3 to 5.99.
The bugs in this list are all unassigned and are ones that I consider to be nice to do some time but not essential track from milestone to milestone in 5.0 or 5.1. We agreed on this bug handling approach in the YP bug review on Feb 22, 2024.

If you disagree, please move back to a 5.[012...]-M* milestone.
Comment 9 Mathieu Dubois-Briand 2024-10-14 06:18:01 UTC 
no-x11 debian12-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/9549/steps/15/logs/stdio
Comment 11 Randy MacLeod 2024-10-24 14:48:51 UTC 
The debian11 issues should be fixed by changing to the 6.1 kernel. 
Waiting to see if debian12 happens again.
Comment 12 Richard Purdie 2024-11-02 22:05:37 UTC 
No recurrences reported so closing.