| Summary: | Occasional ssh timeouts | ||
|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Ross Burton <ross.burton> |
| Component: | oe-selftest | Assignee: | Jon Mason <jon.mason> |
| Status: | RESOLVED WORKSFORME | QA Contact: | |
| Severity: | major | ||
| Priority: | Medium+ | CC: | Harish.Sadineni, jon.mason, randy.macleod, richard.purdie, sundeep.kokkonda, Yash.Shinde |
| Version: | 5.2 | ||
| Target Milestone: | 5.1 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
CC Harish, Yash, Sundeep since they may have a similar problem with ssh for toolchain tests. Jon, Can you describe how you reproduce the retries and what you think could be done to deal with this problem. Likely only happening on arm. Lower importance. Patch submitted to increase the number of attempts https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=c796438eec5dd6b4671b798f85506bc89ff402ab To "debug" this, I found it easier to modify the test and increase the number of attempts to a very high number and add a print to show how many attempts it took when exiting the test. In the patch referenced above, you would modify the attempts (changed to 20). As an example, I used 100 attempts on fvp-base using poky-altcfg, which is very slow emulated system and systemd in poky-altcfg brings up the ssh daemon at a much later point. In this "debug test", it took roughly 70 attempts to pass the ssh test. In this example, increasing the number of attempts isn't an acceptable solution to this issue, as the worst case (which is the test trying 100 times then failing) would be 500 seconds (or 8.33 mins). Taking this long on a failure would cascade a delay in the Autobuilder CI test matrix. So, I need to debug the systemd issue to see why it takes so long (which isn't necessarily relevant to this issue). Still seems to be an issue. RP tried to address it with a recent patch. Keeping this around until it can be verified to go away. Likely fixed by another commit. |
Since the ssh connection changes by Jon we're seeing rare but occasional SSH timeouts: File "/home/pokybuild/yocto-worker/qemumips-alt/build/meta/lib/oeqa/runtime/cases/ssh.py", line 37, in test_ssh self.fail("ssh failed with \"%s\" (exit code %s)" % (output, status)) AssertionError: ssh failed with "" (exit code -15) Potentially we just need to bump the times a little further? https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/6318/steps/14/logs/stdio