Created attachment 4712 [details] do_testimage log for failure https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2304 failed on fedora32-ty-1 for core-image-sato-sdk in do_testimage Its from one of Bruce's kernel test builds but I think unrealated to the kernel changes being tested, its more some kind of ssh connection issue. Adding "import time" to bitbake at the top level hangs bitbake. It shouldn't do that. OTE: test_stap (stap.StapTest) DEBUG: Checking if at least one of systemtap is installed DEBUG: Checking if tools-profile is in DISTRO_FEATURES or IMAGE_FEATURES DEBUG: [Running]$ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR /home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/runtime/files/hello.stp root@192.168.7.6:/tmp/hello.stp DEBUG: Data from SSH call: DEBUG: [Command returned '0' after 0.35 seconds] DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; cd /usr/src/kernel && make scripts prepare DEBUG: time: 1597206070.6914976, endtime: 1597206970.6862304 DEBUG: time: 1597206075.697166, endtime: 1597206970.6862304 DEBUG: time: 1597206080.7024577, endtime: 1597206970.6862304 DEBUG: time: 1597206085.70772, endtime: 1597206970.6862304 DEBUG: time: 1597206090.7129822, endtime: 1597206970.6862304 DEBUG: time: 1597206095.7182498, endtime: 1597206970.6862304 DEBUG: time: 1597206100.7235305, endtime: 1597206970.6862304 DEBUG: [Command returned '0' after 32.33 seconds] DEBUG: Command: cd /usr/src/kernel && make scripts prepare Output: DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; cd /lib/modules/`uname -r` && (if [ ! -e build ]; then ln -s /usr/src/kernel build; fi) DEBUG: time: 1597206103.0190158, endtime: 1597207003.0136237 DEBUG: [Command returned '0' after 0.45 seconds] DEBUG: Command: cd /lib/modules/`uname -r` && (if [ ! -e build ]; then ln -s /usr/src/kernel build; fi) Output: DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp DEBUG: time: 1597206103.4669337, endtime: 1597207003.4614966 DEBUG: Partial data from SSH call: WARNING: Kernel function symbol table missing [man warning::symbols] client_loop: send disconnect: Broken pipe DEBUG: time: 1597214248.0948582, endtime: 1597215148.094855 DEBUG: [Command returned '255' after 8144.63 seconds] DEBUG: Command: stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp Output: WARNING: Kernel function symbol table missing [man warning::symbols] client_loop: send disconnect: Broken pipe DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; rm /tmp/hello.stp DEBUG: time: 1597214248.1033268, endtime: 1597214548.0965524 DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.6 port 22: No route to host DEBUG: time: 1597214251.1667948, endtime: 1597214551.1667922 DEBUG: [Command returned '255' after 3.07 seconds] DEBUG: Command: rm /tmp/hello.stp Output: ssh: connect to host 192.168.7.6 port 22: No route to host NOTE: ... FAIL Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemuarm/build/meta/lib/oeqa/runtime/cases/stap.py", line 37, in test_stap self.assertEqual(status, 0, msg='\n'.join([cmd, output])) AssertionError: 255 != 0 : stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp WARNING: Kernel function symbol table missing [man warning::symbols] client_loop: send disconnect: Broken pipe NOTE: test_systemd_basic (systemd.SystemdBasicTests)
Please ignore the "Adding "import time" to bitbake at the top level hangs bitbake. It shouldn't do that.", that isn't true, a time.sleep(5) in unlockfile() appears to but its just due to the number of lockfiles.
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2321
Starting to investigate basic failure and creating somekind of easier reproducer
Patch submitted for trying to catch the failure and provide more logging details via the TargetDumper. Setting this to NEEDINFO waiting for additional info from the Autobuilder
https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1249/steps/8/logs/step1c occurred which is an ltp no route to host failure. No serial dump?
Randy will find the qemu/monitor enhancement request and one of these bugs can be marked as duplicate. Why didn't we attempt to dump the serial line status?
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/3326
https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/3356/steps/12/logs/stdio master branch on ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/2335/steps/13/logs/stdio qemuarm-alt fedora31-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/2386/steps/13/logs/stdio qemuarm-alt centos7-ty-4
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3552/steps/13/logs/stdio qemuarm64 ubuntu1804-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2894/steps/12/logs/stdio qemuarm64-armhost ubuntu2004-arm-1
Closing as likely caused by rcu stall hangs. Without the kernel logs we can't tell. We'll open new bugs if issues recur.
bulk change to add Saul's non-WR address.
Bulk change: Remove Saul's old WR address.