Bug 14399

Summary: testimage tests fails due to connectivity
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Jon Mason <jon.mason>
Component: virtual machinesAssignee: Alexandre Belloni <alexandre.belloni>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: randy.macleod, richard.purdie
Version: 3.3   
Target Milestone: 3.4   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT AB-INT-TMPFS
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Jon Mason 2021-05-20 14:25:37 UTC 
From https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/3456/steps/12/logs/stdio

Multiple tests fail due to network connectivity issues
Bitbake still alive (5000s)
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/runtime/cases/dnf.py", line 181, in test_dnf_exclude
    self.dnf_with_repo('install -y curl*')
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/runtime/cases/dnf.py", line 69, in dnf_with_repo
    output = self.dnf(" ".join(cmdlinerepoopts) + " --nogpgcheck " + command)
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/runtime/cases/dnf.py", line 21, in dnf
    self.assertEqual(status, expected, message)
AssertionError: 255 != 0 : dnf --repofrompath=oe-testimage-repo-qemuarm,http://192.168.7.1:42885/qemuarm --repofrompath=oe-testimage-repo-noarch,http://192.168.7.1:42885/noarch --repofrompath=oe-testimage-repo-armv7vet2hf_neon,http://192.168.7.1:42885/armv7vet2hf_neon --nogpgcheck install -y curl*
Added oe-testimage-repo-qemuarm repo from http://192.168.7.1:42885/qemuarm
Added oe-testimage-repo-noarch repo from http://192.168.7.1:42885/noarch
Added oe-testimage-repo-armv7vet2hf_neon repo from http://192.168.7.1:42885/armv7vet2hf_neon
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/runtime/cases/dnf.py", line 88, in test_dnf_install
    output = self.dnf_with_repo('list run-postinsts-dev')
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/runtime/cases/dnf.py", line 69, in dnf_with_repo
    output = self.dnf(" ".join(cmdlinerepoopts) + " --nogpgcheck " + command)
  File "/home/pokybuild/yocto-worker/qemuarm-oecore/build/meta/lib/oeqa/runtime/cases/dnf.py", line 21, in dnf
    self.assertEqual(status, expected, message)
AssertionError: 255 != 0 : dnf --repofrompath=oe-testimage-repo-qemuarm,http://192.168.7.1:42885/qemuarm --repofrompath=oe-testimage-repo-noarch,http://192.168.7.1:42885/noarch --repofrompath=oe-testimage-repo-armv7vet2hf_neon,http://192.168.7.1:42885/armv7vet2hf_neon --nogpgcheck list run-postinsts-dev
ssh: connect to host 192.168.7.2 port 22: No route to host

At this point all tests fail due to ssh not being able to connect
Comment 1 Richard Purdie 2021-05-20 20:17:12 UTC 
Took a look at the build directory, only other clue was this in the qemu boot log:

dbus-daemon[580]: Successfully activated service 'org.a11y.atspi.Registry'
SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
matchbox-wm: X error warning (0xa00003): BadWindow (invalid Window parameter) (opcode: 12)

** (matchbox-desktop:593): [1;33mWARNING[0m **: [34m13:07:05.144[0m: Error loading icon: Icon 'net-48d24' not present in theme Sato

** (matchbox-desktop:593): [1;33mWARNING[0m **: [34m13:07:05.200[0m: Error loading icon: Icon 'terminal' not present in theme Sato
[  252.574227] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  252.575402] rcu: 	2-...!: (2 ticks this GP) idle=592/1/0x40000000 softirq=3034/3034 fqs=1 
[  252.576119] 	(detected by 3, t=2505 jiffies, g=5061, q=10)

ssh connection just appeared to die, no saved logs, nothing from qmp
Comment 3 Alexandre Belloni 2021-06-17 14:40:40 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2188/steps/14/logs/stdio

oe-selftest-fedora fedora31-ty-1
Comment 4 Alexandre Belloni 2021-06-20 17:24:26 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2851/steps/12/logs/stdio

qemuarm64-armhost ubuntu2004-arm-1
Comment 6 Richard Purdie 2021-06-28 10:21:18 UTC 
Closing as likely caused by rcu stall hangs. Without the kernel logs we can't tell. We'll open new bugs if issues recur.