15561 2024-07-24 12:56:51 +0000 Occasional ssh timeouts 2024-10-31 15:03:39 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest 5.2 x86 Multiple RESOLVED WORKSFORME Medium+ major 5.1 M4 1 ross.burton jon.mason Harish.Sadineni jon.mason randy.macleod richard.purdie sundeep.kokkonda Yash.Shinde --- --- No (bug/feature does not impact docs) oldest_to_newest 99426 0 ross.burton 2024-07-24 12:56:51 +0000 Since the ssh connection changes by Jon we're seeing rare but occasional SSH timeouts: File "/home/pokybuild/yocto-worker/qemumips-alt/build/meta/lib/oeqa/runtime/cases/ssh.py", line 37, in test_ssh self.fail("ssh failed with \"%s\" (exit code %s)" % (output, status)) AssertionError: ssh failed with "" (exit code -15) Potentially we just need to bump the times a little further? https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/6318/steps/14/logs/stdio 99565 1 randy.macleod 2024-08-22 14:43:58 +0000 CC Harish, Yash, Sundeep since they may have a similar problem with ssh for toolchain tests. Jon, Can you describe how you reproduce the retries and what you think could be done to deal with this problem. 99566 2 randy.macleod 2024-08-22 14:56:05 +0000 Likely only happening on arm. Lower importance. 99569 3 jon.mason 2024-08-23 14:49:02 +0000 Patch submitted to increase the number of attempts https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=c796438eec5dd6b4671b798f85506bc89ff402ab To "debug" this, I found it easier to modify the test and increase the number of attempts to a very high number and add a print to show how many attempts it took when exiting the test. In the patch referenced above, you would modify the attempts (changed to 20). As an example, I used 100 attempts on fvp-base using poky-altcfg, which is very slow emulated system and systemd in poky-altcfg brings up the ssh daemon at a much later point. In this "debug test", it took roughly 70 attempts to pass the ssh test. In this example, increasing the number of attempts isn't an acceptable solution to this issue, as the worst case (which is the test trying 100 times then failing) would be 500 seconds (or 8.33 mins). Taking this long on a failure would cascade a delay in the Autobuilder CI test matrix. So, I need to debug the systemd issue to see why it takes so long (which isn't necessarily relevant to this issue). 99963 4 jon.mason 2024-10-22 13:33:38 +0000 Still seems to be an issue. RP tried to address it with a recent patch. Keeping this around until it can be verified to go away. 100070 5 randy.macleod 2024-10-31 15:03:39 +0000 Likely fixed by another commit.