| Summary: | AB-INT: systemd.SystemdServiceTests.test_systemd_status failure | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Alexandre Belloni <alexandre.belloni> | ||||||||||||
| Component: | testimage | Assignee: | Ross Burton <ross.burton> | ||||||||||||
| Status: | RESOLVED OBSOLETE | QA Contact: | |||||||||||||
| Severity: | normal | ||||||||||||||
| Priority: | Medium+ | CC: | alex.kanavin, alexandre.belloni, JPEWhacker, luca.ceresoli, pavel, randy.macleod, richard.purdie, ross.burton, steve, tgamblin | ||||||||||||
| Version: | unspecified | ||||||||||||||
| Target Milestone: | 5.2 M3 | ||||||||||||||
| Hardware: | x86 | ||||||||||||||
| OS: | Multiple | ||||||||||||||
| Whiteboard: | AB-INT AB-FREQUENT | ||||||||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||||||||
| Attachments: |
|
||||||||||||||
|
Description
Alexandre Belloni
2022-04-14 13:46:49 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3953/steps/13/logs/stdio qemuarm-alt debian11-ty-1 Here's the log from the second fail: https://autobuilder.yocto.io/pub/failed-builds-data/debian11-20220415-qemuarm/ The trouble seem to start during dnf tests where run-postinsts is repeatedly installed and deinstalled, issuing systemctl commands: %post(run-postinsts-1.0-r10.noarch): scriptlet start %post(run-postinsts-1.0-r10.noarch): execv(/bin/sh) pid 11395 + set -e + systemctl + OPTS= + '[' -n '' ']' + '[' enable = enable ']' + for service in run-postinsts.service + systemctl enable run-postinsts.service Created symlink /etc/systemd/system/sysinit.target.wants/run-postinsts.service → /lib/systemd/system/run-postinsts.service. + '[' -z '' ']' + systemctl daemon-reload + systemctl preset run-postinsts.service + '[' enable = enable ']' + systemctl --no-block restart run-postinsts.service %post(run-postinsts-1.0-r10.noarch): waitpid(11395) rc 11395 status 0 At some point during this, systemd catastrophically fails and won't restart some services, including avahi, which leads to the observed test failure later on: Apr 14 22:22:08 qemuarm systemd-logind[219]: New session c46 of user root. Apr 14 22:22:08 qemuarm systemd[1]: Started Session c46 of User root. Apr 14 22:22:12 qemuarm systemd-journald[159]: Forwarding to syslog missed 23 messages. Apr 14 22:22:20 qemuarm systemd[1]: Reloading. Apr 14 22:22:27 qemuarm systemd[1]: Reloading. Apr 14 22:22:37 qemuarm systemd[1]: Reloading. Apr 14 22:23:09 qemuarm systemd[1]: Starting Run pending postinsts... Apr 14 22:23:09 qemuarm systemd-journald[159]: Forwarding to syslog missed 4 messages. Apr 14 22:23:11 qemuarm systemd[1]: systemd-timesyncd.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: systemd-resolved.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: systemd-networkd.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: systemd-logind.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: ofono.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: connman.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: avahi-daemon.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: wpa_supplicant.service: Unexpected error response from GetNameOwner(): Connection terminated Apr 14 22:23:11 qemuarm systemd[1]: wpa_supplicant.service: Deactivated successfully. Apr 14 22:23:11 qemuarm run-postinsts[11437]: Removing any system startup links for run-postinsts ... Apr 14 22:23:11 qemuarm systemd[1]: systemd-logind.service: Deactivated successfully. Apr 14 22:23:11 qemuarm systemd[1]: avahi-daemon.service: Deactivated successfully. Apr 14 22:23:12 qemuarm systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1. Apr 14 22:23:12 qemuarm systemd[1]: Stopped User Login Management. Apr 14 22:23:12 qemuarm systemd[1]: Starting Load Kernel Module drm... Apr 14 22:23:12 qemuarm systemd[1]: modprobe@drm.service: Deactivated successfully. Apr 14 22:23:12 qemuarm systemd[1]: Finished Load Kernel Module drm. Apr 14 22:23:15 qemuarm systemctl[11441]: Removed /etc/systemd/system/sysinit.target.wants/run-postinsts.service. Apr 14 22:23:16 qemuarm systemd[1]: Starting User Login Management... Apr 14 22:23:16 qemuarm systemd[1]: Finished Run pending postinsts. Apr 14 22:23:21 qemuarm systemd[1]: ofono.service: Deactivated successfully. Apr 14 22:23:28 qemuarm systemd[1]: connman.service: Deactivated successfully. Apr 14 22:23:29 qemuarm systemd-logind[11444]: New seat seat0. Apr 14 22:23:29 qemuarm systemd-logind[11444]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) Apr 14 22:23:29 qemuarm systemd[1]: Started User Login Management. Apr 14 22:23:29 qemuarm systemd-logind[11444]: New session c1 of user root. Apr 14 22:23:29 qemuarm systemd-logind[11444]: New session c46 of user root. Apr 14 22:23:31 qemuarm systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.7.3:47126). Apr 14 22:23:31 qemuarm systemd[1]: sshd@44-192.168.7.4:22-192.168.7.3:47124.service: Deactivated successfully. Apr 14 22:23:31 qemuarm systemd[1]: session-c46.scope: Deactivated successfully. Apr 14 22:23:31 qemuarm systemd-logind[11444]: Session c46 logged out. Waiting for processes to exit. Apr 14 22:23:31 qemuarm systemd-logind[11444]: Removed session c46. 'daemon-reload' seems to be a valid command to use during package installs, and both fedora and debian use it (although it's hard to tell for sure because they hide it behind abstractions and macros). My best theory is that non-kvm test runs expose a timing bug in systemd somehow. Joshua has some ideas so I've CCed him. Created attachment 4862 [details]
Script to reload systemd
Created attachment 4863 [details]
Test dbus service that never starts
My theory is that there is a race in systemd when it gets reloaded while waiting on a service to create a dbus name (which is what avahi is doing). I used the attached script and service file to try and reproduce this in a Fedora docker container, going all the way back to Fedora 32 (as an easy way of trying multiple versions of systemd), but was unable to reproduce it. It might be worth trying the attached files to reproduce on hardware; install the dbus service as test.service, then run the reload script while it is starting. New failure, qemumips-alt, debian9-ty-2: https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/2971 Log at: https://autobuilder.yocto.io/pub/failed-builds-data/denian9-ty-2-qemumips-alt-20220427/log.do_testimage.28429 The latest one looks different, both avahi and wpa_supplicant seem to fail on startup due to what looks like internal timeouts: Apr 26 22:59:21 qemumips systemd[1]: avahi-daemon.service: Main process exited, code=exited, status=255/EXCEPTION Apr 26 22:59:21 qemumips systemd[1]: avahi-daemon.service: Failed with result 'exit-code'. Apr 26 22:59:21 qemumips systemd[1]: Failed to start Avahi mDNS/DNS-SD Stack. Apr 26 22:59:32 qemumips wpa_supplicant[185]: dbus: Could not acquire the system bus: org.freedesktop.DBus.Error.NoReply - Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. Apr 26 22:59:32 qemumips wpa_supplicant[185]: Failed to initialize wpa_supplicant Apr 26 22:59:32 qemumips systemd[1]: wpa_supplicant.service: Main process exited, code=exited, status=255/EXCEPTION Apr 26 22:59:32 qemumips systemd[1]: wpa_supplicant.service: Failed with result 'exit-code'. Apr 26 22:59:32 qemumips systemd[1]: Failed to start WPA supplicant. An overloaded host in this case, perhaps? kirkstone: https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3026 qemumips-alt fedora34-ty-1 no SSD https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3053/steps/13/logs/stdio qemumips-alt on debian9-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3064/steps/12/logs/stdio qemumips-alt on fedora34-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3334/steps/13/logs/stdio qemumips-alt on opensuse154-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/4660/steps/13/logs/stdio qemuarm-alt fedora36-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/4719/steps/13/logs/stdio qemuarm-alt fedora36-ty-1 Created attachment 4902 [details]
log.do_testimage
Created attachment 4903 [details]
qemu_boot_log
Created attachment 4904 [details]
dmesg from the host
It looks like Avahi (and several other services) die much earlier in the testing sequence, but it's not detected until later. Looking at the logs, there is this sequence which is the output from `journalctl`: Sep 08 01:56:05 qemuarm systemd[1]: Reloading. Sep 08 01:56:22 qemuarm systemd[1]: Reloading. Sep 08 01:56:22 qemuarm systemd-journald[162]: Forwarding to syslog missed 10 messages. Sep 08 01:56:38 qemuarm systemd[1]: Reloading. Sep 08 01:57:23 qemuarm systemd-journald[162]: Forwarding to syslog missed 1 messages. Sep 08 01:57:23 qemuarm systemd[1]: systemd-timesyncd.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: systemd-resolved.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: systemd-networkd.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: systemd-logind.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: ofono.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: connman.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: avahi-daemon.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: wpa_supplicant.service: Unexpected error response from GetNameOwner(): Connection terminated Sep 08 01:57:23 qemuarm systemd[1]: systemd-logind.service: Deactivated successfully. Sep 08 01:57:24 qemuarm systemd[1]: avahi-daemon.service: Deactivated successfully. Sep 08 01:57:24 qemuarm systemd[1]: wpa_supplicant.service: Deactivated successfully. Sep 08 01:57:24 qemuarm systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1. Sep 08 01:57:24 qemuarm systemd[1]: Stopped User Login Management. Sep 08 01:57:24 qemuarm systemd[1]: Starting Load Kernel Module drm... Sep 08 01:57:24 qemuarm systemd[1]: Starting Run pending postinsts... We can see that avahi-daemon.service is being deactivated around 1:57:23; it's unclear why GetNameOwner() is failing, but I think this might be a red-herring; it shows up in the logs, but avahi-daemon.service is actually just being shutdown completely and that error happens to also occur (my guess is that d-bus is restarting at the same time these services are shutting down). What we would like to know is which test was actually running when avahi was deactivated, so I wrote a simple python program that will trawl the logs for the "DEBUG: time:" messages to figure out which one occurs before 1:57:23, and thus is likely the culprit. First, though we need to convert the time base to the Unix Epoch timestamps used by the DEBUG log messages. To do this, we a correlation timestamp, which we can roughly get from when QEMU boots; early in the logs we see: DEBUG: Reached login banner in 47.49084496498108 seconds (09/08/22 01:11:14) ... NOTE: test_ssh (ssh.SSHTest) DEBUG: Checking if at least one of dropbear, openssh-sshd is installed DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; uname -a DEBUG: time: 1662599499.6276824, endtime: 1662599799.6263304 So we can _roughly_ assume that 1662599499 is 1:11:14, or at least close enough for our purposes. It's probably shifted a little more than that, but should be within a few seconds. Since avahi dies at 1:57:23, we add (1:57:23 - 1:11:14 =~ 2769) seconds to our time base to get 1662602259 as our target log time. My program reveals that this occurs 106 seconds after line 3979 in the log. This line is the command: ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.6 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; dnf --repofrompath=oe-testimage-repo-cortexa15t2hf_neon,http://192.168.7.5:33023/cortexa15t2hf_neon --repofrompath=oe-testimage-repo-qemuarm,http://192.168.7.5:33023/qemuarm --repofrompath=oe-testimage-repo-noarch,http://192.168.7.5:33023/noarch --nogpgcheck install -y /var/cache/dnf/oe-testimage-repo-noarch-ff89a9831e61bcde/packages/run-postinsts-dev-1.0-r10.noarch.rpm Which is part of the test_dnf_install_from_disk test. My guess is that somehow the post-install scripts are stopping some services, either by accident or on purpose. https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/3595/steps/12/logs/stdio qemumips-alt alma9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/3571/steps/13/logs/stdio qemumips-alt fedora36-ty-1 Patches on the list to switch run-postinsts to new dnf-test-* packages. Hopefully fixed with fb1de2abc53bd742bc55cfecd384b78852c10d80. qemumips-alt still seems to fail: https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/3708/steps/12/logs/stdio qemumips-alt ubuntu1804-ty-3 https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/3688/steps/12/logs/stdio qemumips-alt ubuntu2204-ty-2 Hopefully fixed, hasn't replicated since. https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/4070/steps/13/logs/stdio qemumips-alt debian11-ty-2 As mentioned likely fixed by: https://git.openembedded.org/openembedded-core/commit/?id=fb1de2abc53bd742bc55cfecd384b78852c10d80 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/6278/steps/13/logs/stdio qemuarm-alt debian11-ty-2 Note that the suspicious rpm trace is fixed in 017f3a0b1265 ("oeqa/runtime/cases/rpm: fix wait_for_no_process_for_user failure case") Moved to M4. Not seen in over a year, assuming addressed in new versions of systemd. |