Bug 14787 - AB-INT: systemd.SystemdServiceTests.test_systemd_status failure
Summary: AB-INT: systemd.SystemdServiceTests.test_systemd_status failure
Status: RESOLVED OBSOLETE
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: testimage (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.2 M3
Assignee: Ross Burton
QA Contact:
URL:
Whiteboard: AB-INT AB-FREQUENT
Depends on:
Blocks:
 
Reported: 2022-04-14 13:46 UTC by Alexandre Belloni
Modified: 2024-12-19 15:53 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Script to reload systemd (150 bytes, text/plain)
2022-04-26 15:35 UTC, Joshua Watt
no flags Details
Test dbus service that never starts (142 bytes, text/x-dbus-service)
2022-04-26 15:36 UTC, Joshua Watt
no flags Details
log.do_testimage (1.39 MB, text/plain)
2022-09-08 21:45 UTC, Alexandre Belloni
no flags Details
qemu_boot_log (18.74 KB, text/plain)
2022-09-08 21:46 UTC, Alexandre Belloni
no flags Details
dmesg from the host (115.18 KB, text/plain)
2022-09-08 22:01 UTC, Alexandre Belloni
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alexandre Belloni 2022-04-14 13:46:49 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/2879/steps/13/logs/stdio

qemuppc-alt centos8-ty-1 no SSD

Startup finished in 6.837s (kernel) + 40.879s (userspace) = 47.717s.
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuppc-alt/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuppc-alt/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuppc-alt/build/meta/lib/oeqa/runtime/cases/systemd.py", line 106, in test_systemd_status
    self.systemctl('status --full', 'avahi-daemon.service')
  File "/home/pokybuild/yocto-worker/qemuppc-alt/build/meta/lib/oeqa/runtime/cases/systemd.py", line 23, in systemctl
    self.assertEqual(status, expected, message)
AssertionError: 3 != 0 : SYSTEMD_BUS_TIMEOUT=240s systemctl status --full avahi-daemon.service
○ avahi-daemon.service - Avahi mDNS/DNS-SD Stack
     Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Fri 2022-04-08 22:22:17 UTC; 18min ago
TriggeredBy: ● avahi-daemon.socket
    Process: 183 ExecStart=/usr/sbin/avahi-daemon -s (code=exited, status=0/SUCCESS)
   Main PID: 183 (code=exited, status=0/SUCCESS)
     Status: "avahi-daemon 0.8 starting up."
Apr 08 21:51:56 qemuppc systemd[1]: Starting Avahi mDNS/DNS-SD Stack...
Apr 08 21:51:59 qemuppc systemd[1]: Started Avahi mDNS/DNS-SD Stack.
Apr 08 22:22:15 qemuppc systemd[1]: avahi-daemon.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 08 22:22:17 qemuppc systemd[1]: avahi-daemon.service: Deactivated successfully.
Comment 1 Richard Purdie 2022-04-15 16:10:01 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3953/steps/13/logs/stdio qemuarm-alt debian11-ty-1
Comment 2 Alexander Kanavin 2022-04-15 18:01:52 UTC
Here's the log from the second fail:
https://autobuilder.yocto.io/pub/failed-builds-data/debian11-20220415-qemuarm/

The trouble seem to start during dnf tests where run-postinsts is repeatedly installed and deinstalled, issuing systemctl commands:

%post(run-postinsts-1.0-r10.noarch): scriptlet start
%post(run-postinsts-1.0-r10.noarch): execv(/bin/sh) pid 11395
+ set -e
+ systemctl
+ OPTS=
+ '[' -n '' ']'
+ '[' enable = enable ']'
+ for service in run-postinsts.service
+ systemctl enable run-postinsts.service
Created symlink /etc/systemd/system/sysinit.target.wants/run-postinsts.service → /lib/systemd/system/run-postinsts.service.
+ '[' -z '' ']'
+ systemctl daemon-reload
+ systemctl preset run-postinsts.service
+ '[' enable = enable ']'
+ systemctl --no-block restart run-postinsts.service
%post(run-postinsts-1.0-r10.noarch): waitpid(11395) rc 11395 status 0

At some point during this, systemd catastrophically fails and won't restart some services, including avahi, which leads to the observed test failure later on:

Apr 14 22:22:08 qemuarm systemd-logind[219]: New session c46 of user root.
Apr 14 22:22:08 qemuarm systemd[1]: Started Session c46 of User root.
Apr 14 22:22:12 qemuarm systemd-journald[159]: Forwarding to syslog missed 23 messages.
Apr 14 22:22:20 qemuarm systemd[1]: Reloading.
Apr 14 22:22:27 qemuarm systemd[1]: Reloading.
Apr 14 22:22:37 qemuarm systemd[1]: Reloading.
Apr 14 22:23:09 qemuarm systemd[1]: Starting Run pending postinsts...
Apr 14 22:23:09 qemuarm systemd-journald[159]: Forwarding to syslog missed 4 messages.
Apr 14 22:23:11 qemuarm systemd[1]: systemd-timesyncd.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: systemd-resolved.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: systemd-networkd.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: systemd-logind.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: ofono.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: connman.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: avahi-daemon.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: wpa_supplicant.service: Unexpected error response from GetNameOwner(): Connection terminated
Apr 14 22:23:11 qemuarm systemd[1]: wpa_supplicant.service: Deactivated successfully.
Apr 14 22:23:11 qemuarm run-postinsts[11437]:  Removing any system startup links for run-postinsts ...
Apr 14 22:23:11 qemuarm systemd[1]: systemd-logind.service: Deactivated successfully.
Apr 14 22:23:11 qemuarm systemd[1]: avahi-daemon.service: Deactivated successfully.
Apr 14 22:23:12 qemuarm systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
Apr 14 22:23:12 qemuarm systemd[1]: Stopped User Login Management.
Apr 14 22:23:12 qemuarm systemd[1]: Starting Load Kernel Module drm...
Apr 14 22:23:12 qemuarm systemd[1]: modprobe@drm.service: Deactivated successfully.
Apr 14 22:23:12 qemuarm systemd[1]: Finished Load Kernel Module drm.
Apr 14 22:23:15 qemuarm systemctl[11441]: Removed /etc/systemd/system/sysinit.target.wants/run-postinsts.service.
Apr 14 22:23:16 qemuarm systemd[1]: Starting User Login Management...
Apr 14 22:23:16 qemuarm systemd[1]: Finished Run pending postinsts.
Apr 14 22:23:21 qemuarm systemd[1]: ofono.service: Deactivated successfully.
Apr 14 22:23:28 qemuarm systemd[1]: connman.service: Deactivated successfully.
Apr 14 22:23:29 qemuarm systemd-logind[11444]: New seat seat0.
Apr 14 22:23:29 qemuarm systemd-logind[11444]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard)
Apr 14 22:23:29 qemuarm systemd[1]: Started User Login Management.
Apr 14 22:23:29 qemuarm systemd-logind[11444]: New session c1 of user root.
Apr 14 22:23:29 qemuarm systemd-logind[11444]: New session c46 of user root.
Apr 14 22:23:31 qemuarm systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.7.3:47126).
Apr 14 22:23:31 qemuarm systemd[1]: sshd@44-192.168.7.4:22-192.168.7.3:47124.service: Deactivated successfully.
Apr 14 22:23:31 qemuarm systemd[1]: session-c46.scope: Deactivated successfully.
Apr 14 22:23:31 qemuarm systemd-logind[11444]: Session c46 logged out. Waiting for processes to exit.
Apr 14 22:23:31 qemuarm systemd-logind[11444]: Removed session c46.
Comment 3 Alexander Kanavin 2022-04-15 18:08:54 UTC
'daemon-reload' seems to be a valid command to use during package installs, and both fedora and debian use it (although it's hard to tell for sure because they hide it behind abstractions and macros). My best theory is that non-kvm test runs expose a timing bug in systemd somehow.
Comment 4 Randy MacLeod 2022-04-21 15:08:31 UTC
Joshua has some ideas so I've CCed him.
Comment 5 Joshua Watt 2022-04-26 15:35:58 UTC
Created attachment 4862 [details]
Script to reload systemd
Comment 6 Joshua Watt 2022-04-26 15:36:33 UTC
Created attachment 4863 [details]
Test dbus service that never starts
Comment 7 Joshua Watt 2022-04-26 15:39:50 UTC
My theory is that there is a race in systemd when it gets reloaded while waiting on a service to create a dbus name (which is what avahi is doing).

I used the attached script and service file to try and reproduce this in a Fedora docker container, going all the way back to Fedora 32 (as an easy way of trying multiple versions of systemd), but was unable to reproduce it.

It might be worth trying the attached files to reproduce on hardware; install the dbus service as test.service, then run the reload script while it is starting.
Comment 9 Alexander Kanavin 2022-04-27 19:04:52 UTC
The latest one looks different, both avahi and wpa_supplicant seem to fail on startup due to what looks like internal timeouts:

Apr 26 22:59:21 qemumips systemd[1]: avahi-daemon.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 26 22:59:21 qemumips systemd[1]: avahi-daemon.service: Failed with result 'exit-code'.
Apr 26 22:59:21 qemumips systemd[1]: Failed to start Avahi mDNS/DNS-SD Stack.

Apr 26 22:59:32 qemumips wpa_supplicant[185]: dbus: Could not acquire the system bus: org.freedesktop.DBus.Error.NoReply - Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Apr 26 22:59:32 qemumips wpa_supplicant[185]: Failed to initialize wpa_supplicant
Apr 26 22:59:32 qemumips systemd[1]: wpa_supplicant.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 26 22:59:32 qemumips systemd[1]: wpa_supplicant.service: Failed with result 'exit-code'.
Apr 26 22:59:32 qemumips systemd[1]: Failed to start WPA supplicant.


An overloaded host in this case, perhaps?
Comment 10 Steve Sakoman 2022-05-09 19:13:54 UTC
kirkstone:

https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3026

qemumips-alt fedora34-ty-1 no SSD
Comment 11 Luca Ceresoli 2022-05-16 12:16:23 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3053/steps/13/logs/stdio

qemumips-alt on debian9-ty-2
Comment 12 Luca Ceresoli 2022-05-19 13:45:27 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3064/steps/12/logs/stdio

qemumips-alt on fedora34-ty-1
Comment 13 Luca Ceresoli 2022-07-25 19:00:45 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3334/steps/13/logs/stdio

qemumips-alt on opensuse154-ty-2
Comment 14 Alexandre Belloni 2022-09-01 19:11:16 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/4660/steps/13/logs/stdio

qemuarm-alt fedora36-ty-2
Comment 16 Alexandre Belloni 2022-09-08 21:31:42 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/4719/steps/13/logs/stdio

qemuarm-alt fedora36-ty-1
Comment 17 Alexandre Belloni 2022-09-08 21:45:34 UTC
Created attachment 4902 [details]
log.do_testimage
Comment 18 Alexandre Belloni 2022-09-08 21:46:10 UTC
Created attachment 4903 [details]
qemu_boot_log
Comment 19 Alexandre Belloni 2022-09-08 22:01:09 UTC
Created attachment 4904 [details]
dmesg from the host
Comment 20 Joshua Watt 2022-09-09 14:18:25 UTC
It looks like Avahi (and several other services) die much earlier in the testing sequence, but it's not detected until later. Looking at the logs, there is this sequence which is the output from `journalctl`:


Sep 08 01:56:05 qemuarm systemd[1]: Reloading.
Sep 08 01:56:22 qemuarm systemd[1]: Reloading.
Sep 08 01:56:22 qemuarm systemd-journald[162]: Forwarding to syslog missed 10 messages.
Sep 08 01:56:38 qemuarm systemd[1]: Reloading.
Sep 08 01:57:23 qemuarm systemd-journald[162]: Forwarding to syslog missed 1 messages.
Sep 08 01:57:23 qemuarm systemd[1]: systemd-timesyncd.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: systemd-resolved.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: systemd-networkd.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: systemd-logind.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: ofono.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: connman.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: avahi-daemon.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: wpa_supplicant.service: Unexpected error response from GetNameOwner(): Connection terminated
Sep 08 01:57:23 qemuarm systemd[1]: systemd-logind.service: Deactivated successfully.
Sep 08 01:57:24 qemuarm systemd[1]: avahi-daemon.service: Deactivated successfully.
Sep 08 01:57:24 qemuarm systemd[1]: wpa_supplicant.service: Deactivated successfully.
Sep 08 01:57:24 qemuarm systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
Sep 08 01:57:24 qemuarm systemd[1]: Stopped User Login Management.
Sep 08 01:57:24 qemuarm systemd[1]: Starting Load Kernel Module drm...
Sep 08 01:57:24 qemuarm systemd[1]: Starting Run pending postinsts...


We can see that avahi-daemon.service is being deactivated around 1:57:23; it's unclear why GetNameOwner() is failing, but I think this might be a red-herring; it shows up in the logs, but avahi-daemon.service is actually just being shutdown completely and that error happens to also occur (my guess is that d-bus is restarting at the same time these services are shutting down).

What we would like to know is which test was actually running when avahi was deactivated, so I wrote a simple python program that will trawl the logs for the "DEBUG: time:" messages to figure out which one occurs before 1:57:23, and thus is likely the culprit.

First, though we need to convert the time base to the Unix Epoch timestamps used by the DEBUG log messages. To do this, we a correlation timestamp, which we can roughly get from when QEMU boots; early in the logs we see: 

DEBUG: Reached login banner in 47.49084496498108 seconds (09/08/22 01:11:14)
...
NOTE: test_ssh (ssh.SSHTest)
DEBUG: Checking if at least one of dropbear, openssh-sshd is installed
DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; uname -a
DEBUG: time: 1662599499.6276824, endtime: 1662599799.6263304

So we can _roughly_ assume that 1662599499 is 1:11:14, or at least close enough for our purposes. It's probably shifted a little more than that, but should be within a few seconds.

Since avahi dies at 1:57:23, we add (1:57:23 - 1:11:14 =~ 2769) seconds to our time base to get 1662602259 as our target log time.

My program reveals that this occurs 106 seconds after line 3979 in the log. This line is the command:

ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; dnf --repofrompath=oe-testimage-repo-cortexa15t2hf_neon,http://192.168.7.5:33023/cortexa15t2hf_neon --repofrompath=oe-testimage-repo-qemuarm,http://192.168.7.5:33023/qemuarm --repofrompath=oe-testimage-repo-noarch,http://192.168.7.5:33023/noarch --nogpgcheck install -y /var/cache/dnf/oe-testimage-repo-noarch-ff89a9831e61bcde/packages/run-postinsts-dev-1.0-r10.noarch.rpm

Which is part of the test_dnf_install_from_disk test.

My guess is that somehow the post-install scripts are stopping some services, either by accident or on purpose.
Comment 21 Alexandre Belloni 2022-09-22 12:04:18 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3595/steps/12/logs/stdio

qemumips-alt alma9-ty-1
Comment 22 Alexandre Belloni 2022-09-22 13:22:35 UTC
https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/3571/steps/13/logs/stdio

qemumips-alt fedora36-ty-1
Comment 23 Ross Burton 2022-09-26 11:16:45 UTC
Patches on the list to switch run-postinsts to new dnf-test-* packages.
Comment 24 Ross Burton 2022-09-28 14:39:50 UTC
Hopefully fixed with fb1de2abc53bd742bc55cfecd384b78852c10d80.
Comment 25 Alexandre Belloni 2022-11-02 20:01:21 UTC
qemumips-alt still seems to fail:

https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/3708/steps/12/logs/stdio

qemumips-alt ubuntu1804-ty-3

https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/3688/steps/12/logs/stdio

qemumips-alt ubuntu2204-ty-2
Comment 26 Ross Burton 2022-11-24 15:56:18 UTC
Hopefully fixed, hasn't replicated since.
Comment 27 Alexandre Belloni 2023-02-16 16:48:57 UTC
https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/4070/steps/13/logs/stdio

qemumips-alt debian11-ty-2
Comment 29 Alexandre Belloni 2023-07-19 13:02:49 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/6278/steps/13/logs/stdio

qemuarm-alt debian11-ty-2


Note that the suspicious rpm trace is fixed in 017f3a0b1265 ("oeqa/runtime/cases/rpm: fix wait_for_no_process_for_user failure case")
Comment 30 Trevor Gamblin 2023-09-14 15:33:12 UTC
Moved to M4.
Comment 31 Richard Purdie 2024-12-19 15:53:19 UTC
Not seen in over a year, assuming addressed in new versions of systemd.