Bug 15620 - AB-INT: AssertionError: Ping timeout error for address 192.168.7.x
Summary: AB-INT: AssertionError: Ping timeout error for address 192.168.7.x
Status: RESOLVED FIXED
Alias: None
Product: Functional (self) Testing
Classification: QA/Testing
Component: oe-selftest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High normal
Target Milestone: 5.2 M1
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard: AB-INT
Depends on:
Blocks:
 
Reported: 2024-10-14 06:15 UTC by Mathieu Dubois-Briand
Modified: 2024-10-23 14:48 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Dubois-Briand 2024-10-14 06:15:15 UTC
Saw several similar issues on the autobuilder in the last few days:

I'm not quite sure they all have the same root cause. Also, it might be related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=15327.

Related message on ML: https://lists.openembedded.org/g/openembedded-core/topic/latest_ab_int_unexplained/108984574

oe-selftest debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/69/builds/15/steps/14/logs/stdio

QMP Available for connection at /srv/pokybuild/yocto-worker/oe-selftest/build/build-st-2146403/tmp/.sxwesxl5
QMP connected to QEMU at 10/11/24 03:34:58 and took 0.56 seconds
QMP released QEMU at 10/11/24 03:34:59 and took 0.10 seconds from connect
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest/build/meta/lib/oeqa/runtime/cases/ping.py", line 25, in test_ping
    output += proc.communicate()[0].decode('utf-8')
  File "/usr/lib/python3.9/subprocess.py", line 1121, in communicate
    stdout = self.stdout.read()
oeqa.core.exception.OEQATimeoutError: Timed out after 30 seconds of execution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest/build/meta/lib/oeqa/runtime/cases/ping.py", line 32, in test_ping
    self.fail("Ping timeout error for address %s, count %s, output: %s" % (self.target.ip, count, output))
AssertionError: Ping timeout error for address 192.168.7.4, count 0, output: PING 192.168.7.4 (192.168.7.4) 56(84) bytes of data.
---

oe-selftest-debian debian11-vk-1
https://valkyrie.yoctoproject.org/#/builders/35/builds/216/steps/14/logs/stdio

QMP released QEMU at 10/13/24 03:43:10 and took 0.14 seconds from connect
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/apt.py", line 76, in test_apt_install_from_repo
    self.pkg('update')
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/apt.py", line 19, in pkg
    self.assertEqual(status, expected, message)
AssertionError: 255 != 0 : apt-get update
ssh: connect to host 192.168.7.2 port 22: No route to host 

Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/ping.py", line 25, in test_ping
    output += proc.communicate()[0].decode('utf-8')
  File "/usr/lib/python3.9/subprocess.py", line 1121, in communicate
    stdout = self.stdout.read()
oeqa.core.exception.OEQATimeoutError: Timed out after 30 seconds of execution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/runtime/cases/ping.py", line 32, in test_ping
    self.fail("Ping timeout error for address %s, count %s, output: %s" % (self.target.ip, count, output))
AssertionError: Ping timeout error for address 192.168.7.2, count 0, output: PING 192.168.7.2 (192.168.7.2) 56(84) bytes of data.
From 192.168.7.1 icmp_seq=1 Destination Host Unreachable
Comment 1 Mathieu Dubois-Briand 2024-10-15 16:27:55 UTC
See original email in this thread:
https://lists.openembedded.org/g/openembedded-core/message/205723
Comment 2 Mathieu Dubois-Briand 2024-10-16 15:45:31 UTC
oe-selftest-debian debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/35/builds/237/steps/14/logs/stdio
Comment 3 Mathieu Dubois-Briand 2024-10-17 14:23:39 UTC
oe-selftest debian11-vk-2
https://valkyrie.yoctoproject.org/#/builders/69/builds/20/steps/14/logs/stdio
Comment 4 Randy MacLeod 2024-10-17 14:39:25 UTC
Not seen on scarthgap on the new autobuilder but almost always happens in builds and only for debian11 hosts.

CCing: Robert and Hongxu since they have been doing debian work recently.
There are several tun/tap network-related symptoms (timeout, no route, more)...
Comment 6 Mathieu Dubois-Briand 2024-10-23 08:45:07 UTC
oe-selftest-debian debian11-vk-3
https://valkyrie.yoctoproject.org/#/builders/35/builds/295/steps/14/logs/stdio
Comment 7 Richard Purdie 2024-10-23 14:48:44 UTC
There is a discussion on te mailing list about this but the summary is that we upgraded the debian 11 workers to a 6.1 kernel and believe we have not seen this issue since that upgrade.