Bug 15620

Summary: AB-INT: AssertionError: Ping timeout error for address 192.168.7.x
Product: [QA/Testing] Functional (self) Testing Reporter: Mathieu Dubois-Briand <mathieu.dubois-briand>
Component: oe-selftestAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: hongxu.jia, liezhi.yang, randy.macleod, ross.burton
Version: unspecified   
Target Milestone: 5.2 M1   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Mathieu Dubois-Briand 2024-10-14 06:15:15 UTC 
Saw several similar issues on the autobuilder in the last few days:

I'm not quite sure they all have the same root cause. Also, it might be related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=15327.

Related message on ML: https://lists.openembedded.org/g/openembedded-core/topic/latest_ab_int_unexplained/108984574

oe-selftest debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/69/builds/15/steps/14/logs/stdio

QMP Available for connection at /srv/pokybuild/yocto-worker/oe-selftest/build/build-st-2146403/tmp/.sxwesxl5
QMP connected to QEMU at 10/11/24 03:34:58 and took 0.56 seconds
QMP released QEMU at 10/11/24 03:34:59 and took 0.10 seconds from connect
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest/build/meta/lib/oeqa/runtime/cases/ping.py", line 25, in test_ping
    output += proc.communicate()[0].decode('utf-8')
  File "/usr/lib/python3.9/subprocess.py", line 1121, in communicate
    stdout = self.stdout.read()
oeqa.core.exception.OEQATimeoutError: Timed out after 30 seconds of execution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest/build/meta/lib/oeqa/runtime/cases/ping.py", line 32, in test_ping
    self.fail("Ping timeout error for address %s, count %s, output: %s" % (self.target.ip, count, output))
AssertionError: Ping timeout error for address 192.168.7.4, count 0, output: PING 192.168.7.4 (192.168.7.4) 56(84) bytes of data.
---

oe-selftest-debian debian11-vk-1
https://valkyrie.yoctoproject.org/#/builders/35/builds/216/steps/14/logs/stdio

QMP released QEMU at 10/13/24 03:43:10 and took 0.14 seconds from connect
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/apt.py", line 76, in test_apt_install_from_repo
    self.pkg('update')
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/apt.py", line 19, in pkg
    self.assertEqual(status, expected, message)
AssertionError: 255 != 0 : apt-get update
ssh: connect to host 192.168.7.2 port 22: No route to host 

Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/ping.py", line 25, in test_ping
    output += proc.communicate()[0].decode('utf-8')
  File "/usr/lib/python3.9/subprocess.py", line 1121, in communicate
    stdout = self.stdout.read()
oeqa.core.exception.OEQATimeoutError: Timed out after 30 seconds of execution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/ping.py", line 32, in test_ping
    self.fail("Ping timeout error for address %s, count %s, output: %s" % (self.target.ip, count, output))
AssertionError: Ping timeout error for address 192.168.7.2, count 0, output: PING 192.168.7.2 (192.168.7.2) 56(84) bytes of data.
From 192.168.7.1 icmp_seq=1 Destination Host Unreachable
Comment 1 Mathieu Dubois-Briand 2024-10-15 16:27:55 UTC 
See original email in this thread:
https://lists.openembedded.org/g/openembedded-core/message/205723
Comment 2 Mathieu Dubois-Briand 2024-10-16 15:45:31 UTC 
oe-selftest-debian debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/35/builds/237/steps/14/logs/stdio
Comment 3 Mathieu Dubois-Briand 2024-10-17 14:23:39 UTC 
oe-selftest debian11-vk-2
https://valkyrie.yoctoproject.org/#/builders/69/builds/20/steps/14/logs/stdio
Comment 4 Randy MacLeod 2024-10-17 14:39:25 UTC 
Not seen on scarthgap on the new autobuilder but almost always happens in builds and only for debian11 hosts.

CCing: Robert and Hongxu since they have been doing debian work recently.
There are several tun/tap network-related symptoms (timeout, no route, more)...
Comment 6 Mathieu Dubois-Briand 2024-10-23 08:45:07 UTC 
oe-selftest-debian debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/35/builds/295/steps/14/logs/stdio
Comment 7 Richard Purdie 2024-10-23 14:48:44 UTC 
There is a discussion on te mailing list about this but the summary is that we upgraded the debian 11 workers to a 6.1 kernel and believe we have not seen this issue since that upgrade.