Bug 14002

Summary: ssh connection failure during stap testing on qemuarm
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: coreAssignee: Saul Wold <saul.wold>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: alexandre.belloni, jon.mason, meta.mr.watcher, meta.watcher, naveen.kumar.saini, poky.bs.watcher, poky.watcher, randy.macleod, sakib.sajal, sgw, victor.kamensky7
Version: 3.2   
Target Milestone: 3.4   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT AB-INT-TMPFS
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
do_testimage log for failure none

Description Richard Purdie 2020-08-12 02:27:50 UTC
Created attachment 4712 [details]
do_testimage log for failure

https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2304

failed on fedora32-ty-1 for core-image-sato-sdk in do_testimage

Its from one of Bruce's kernel test builds but I think unrealated to the kernel changes being tested, its more some kind of ssh connection issue.

Adding "import time" to bitbake at the top level hangs bitbake. It shouldn't do that.

OTE: test_stap (stap.StapTest)
DEBUG: Checking if at least one of systemtap is installed
DEBUG: Checking if tools-profile is in DISTRO_FEATURES or IMAGE_FEATURES
DEBUG: [Running]$ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR /home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/runtime/files/hello.stp root@192.168.7.6:/tmp/hello.stp
DEBUG: Data from SSH call: 
DEBUG: [Command returned '0' after 0.35 seconds]
DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; cd /usr/src/kernel && make scripts prepare
DEBUG: time: 1597206070.6914976, endtime: 1597206970.6862304
DEBUG: time: 1597206075.697166, endtime: 1597206970.6862304
DEBUG: time: 1597206080.7024577, endtime: 1597206970.6862304
DEBUG: time: 1597206085.70772, endtime: 1597206970.6862304
DEBUG: time: 1597206090.7129822, endtime: 1597206970.6862304
DEBUG: time: 1597206095.7182498, endtime: 1597206970.6862304
DEBUG: time: 1597206100.7235305, endtime: 1597206970.6862304
DEBUG: [Command returned '0' after 32.33 seconds]
DEBUG: Command: cd /usr/src/kernel && make scripts prepare
Output:  

DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; cd /lib/modules/`uname -r` && (if [ ! -e build ]; then ln -s /usr/src/kernel build; fi)
DEBUG: time: 1597206103.0190158, endtime: 1597207003.0136237
DEBUG: [Command returned '0' after 0.45 seconds]
DEBUG: Command: cd /lib/modules/`uname -r` && (if [ ! -e build ]; then ln -s /usr/src/kernel build; fi)
Output:  

DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp
DEBUG: time: 1597206103.4669337, endtime: 1597207003.4614966
DEBUG: Partial data from SSH call: WARNING: Kernel function symbol table missing [man warning::symbols]
client_loop: send disconnect: Broken pipe

DEBUG: time: 1597214248.0948582, endtime: 1597215148.094855
DEBUG: [Command returned '255' after 8144.63 seconds]
DEBUG: Command: stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp
Output:  WARNING: Kernel function symbol table missing [man warning::symbols]
client_loop: send disconnect: Broken pipe

DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; rm /tmp/hello.stp
DEBUG: time: 1597214248.1033268, endtime: 1597214548.0965524
DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.6 port 22: No route to host

DEBUG: time: 1597214251.1667948, endtime: 1597214551.1667922
DEBUG: [Command returned '255' after 3.07 seconds]
DEBUG: Command: rm /tmp/hello.stp
Output:  ssh: connect to host 192.168.7.6 port 22: No route to host

NOTE:  ... FAIL
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/runtime/cases/stap.py", line 37, in test_stap
    self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
AssertionError: 255 != 0 : stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp
WARNING: Kernel function symbol table missing [man warning::symbols]
client_loop: send disconnect: Broken pipe

NOTE: test_systemd_basic (systemd.SystemdBasicTests)
Comment 1 Richard Purdie 2020-08-12 02:39:56 UTC
Please ignore the "Adding "import time" to bitbake at the top level hangs bitbake. It shouldn't do that.", that isn't true, a time.sleep(5) in unlockfile() appears to but its just due to the number of lockfiles.
Comment 2 Richard Purdie 2020-08-15 03:40:08 UTC
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2321
Comment 3 Saul Wold 2020-09-24 08:27:47 UTC
Starting to investigate basic failure and creating somekind of easier reproducer
Comment 4 Saul Wold 2020-10-01 07:45:33 UTC
Patch submitted for trying to catch the failure and provide more logging details via the TargetDumper.

Setting this to NEEDINFO waiting for additional info from the Autobuilder
Comment 5 Richard Purdie 2020-10-09 00:05:24 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1249/steps/8/logs/step1c occurred which is an ltp no route to host failure. No serial dump?
Comment 6 Randy MacLeod 2020-10-15 07:45:15 UTC
Randy will find the qemu/monitor enhancement request and one of these bugs can be marked as duplicate.

Why didn't we attempt to dump the serial line status?
Comment 7 Naveen Saini 2021-04-19 06:39:20 UTC
Happened again:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3326
Comment 8 Alexandre Belloni 2021-04-29 22:25:11 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/3356/steps/12/logs/stdio

master branch on ubuntu1804-ty-3
Comment 10 Alexandre Belloni 2021-06-16 08:02:49 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/2386/steps/13/logs/stdio

qemuarm-alt
centos7-ty-4
Comment 11 Alexandre Belloni 2021-06-19 16:33:30 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3552/steps/13/logs/stdio

qemuarm64 ubuntu1804-ty-2
Comment 12 Alexandre Belloni 2021-06-24 12:26:47 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2894/steps/12/logs/stdio

qemuarm64-armhost ubuntu2004-arm-1
Comment 13 Richard Purdie 2021-06-28 10:22:55 UTC
Closing as likely caused by rcu stall hangs. Without the kernel logs we can't tell. We'll open new bugs if issues recur.
Comment 14 Randy MacLeod 2025-02-14 16:46:23 UTC
bulk change to add Saul's non-WR address.
Comment 15 Randy MacLeod 2025-02-14 16:48:25 UTC
Bulk change: Remove Saul's old WR address.