Bug 8937

Summary: SystemdServiceTest failed on the autobuilder
Product: [Runtime] System Startup Reporter: Saul Wold <sgw>
Component: system-startupAssignee: Aníbal Limón <anibal.limon>
Status: RESOLVED DUPLICATE QA Contact:
Severity: normal    
Priority: Medium+ CC: bluelightning, kevin.tian, liezhi.yang, randy.macleod, sgw, yi.zhao
Version: 2.0   
Target Milestone: 2.1 M4   
Hardware: x86   
OS: Multiple   
URL: http://errors.yoctoproject.org/Errors/Details/27468/
Whiteboard:
OS type for building Yocto: --- Type of Regression: Regression (Used to work)
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Log file from Autobuilder for systemdbasictests failure none

Description Saul Wold 2016-01-15 21:17:42 UTC 
Created attachment 2931 [details]
Log file from Autobuilder for systemdbasictests failure

Build Configuration:
BB_VERSION        = "1.28.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Fedora-21"
TARGET_SYS        = "x86_64-poky-linux"
MACHINE           = "qemux86-64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.0"
TUNE_FEATURES     = "m64 core2"
TARGET_FPU        = ""
meta              
meta-yocto        
meta-yocto-bsp    = "heads/sgw/2.0.1-test:e52406900c0d347a97dcd618aa7773938ea3d836"


| test_systemd_status (oeqa.runtime.systemd.SystemdServiceTests) ... FAIL
| test_systemd_status dump data stored in /tmp/oe-saved-tests/201601151118_test_systemd_status
| test_systemd_stop_start (oeqa.runtime.systemd.SystemdServiceTests) ... skipped 'Testcase dependency not met: test_systemd_status'
| 
| ======================================================================
| FAIL: test_systemd_failed (oeqa.runtime.systemd.SystemdBasicTests)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f
|     return func(*args, **kwargs)
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
|     return f(*args, **kwargs)
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/runtime/systemd.py", line 65, in test_systemd_failed
|     self.assertTrue(match, msg="Some systemd units failed:\n%s" % output)
| AssertionError: None is not true : Some systemd units failed:
|   UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
| ● avahi-daemon.service   loaded failed failed Avahi mDNS/DNS-SD Stack
| ● bluetooth.service      loaded failed failed Bluetooth service
| ● ofono.service          loaded failed failed Telephony service
| ● wpa_supplicant.service loaded failed failed WPA supplicant
| 
| LOAD   = Reflects whether the unit definition was properly loaded.
| ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
| SUB    = The low-level unit activation state, values depend on unit type.
| 
| 4 loaded units listed. Pass --all to see loaded but inactive units, too.
| To show all installed unit files use 'systemctl list-unit-files'.● qemux86-64
|     State: degraded
|      Jobs: 0 queued
|    Failed: 4 units
|     Since: Fri 2016-01-15 10:56:44 UTC; 21min ago
|    CGroup: /
|            ├─ 1 /sbin/init
|            ├─80 /sbin/v86d
|            └─system.slice
|              ��─dbus.service
|              │ └─280 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
|              ├─distcc.service
|              │ ├─495 /usr/bin/distccd --verbose --no-detach --daemon --allow 192.168.7.0/24
|              │ ├─503 /usr/bin/distccd --verbose --no-detach --daemon --allow 192.168.7.0/24
|              │ ├─507 /usr/bin/distccd --verbose --no-detach --daemon --allow 192.168.7.0/24
|              │ └─510 /usr/bin/distccd --verbose --no-detach --daemon --allow 192.168.7.0/24
|              ├─busybox-klogd.service
|              │ └─251 /sbin/klogd -n
|              ├─system-serial\x2dgetty.slice
|              │ ├─serial-getty@ttyS1.service
|              │ │ └─498 -sh
|              │ └─serial-getty@ttyS0.service
|              │   └─497 /sbin/agetty -8 -L --keep-baud ttyS0 115200 xterm
|              ├─systemd-journald.service
|              │ └─78 /lib/systemd/systemd-journald
|              ├─connman.service
|              │ └─430 /usr/sbin/connmand -n
|              ├─systemd-timesyncd.service
|              │ └─662 /lib/systemd/systemd-timesyncd
|              ├─systemd-logind.service
|              │ └─499 /lib/systemd/systemd-logind
|              ├─system-getty.slice
|              │ └─getty@tty1.service
|              │   └─275 /sbin/agetty --noclear tty1 linux
|              ├─systemd-udevd.service
|              │ └─101 /lib/systemd/systemd-udevd
|              ├─busybox-syslog.service
|              │ └─582 /sbin/syslogd -n -C
|              ├─system-dropbear.slice
|              │ ├─dropbear@13-192.168.7.6:22-192.168.7.5:37914.service
|              │ │ └─617 x11vnc -display :0 -bg -o x11vnc.log
|              │ └─dropbear@135-192.168.7.6:22-192.168.7.5:38055.service
|              │   ├─1773 /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
|              │   ├─1774 sh -c  . /etc/profile; systemctl status --full --failed
|              │   └─1775 systemctl status --full --failed
|              └─xserver-nodm.service
|                ├─264 xinit /etc/X11/Xsession -- /usr/bin/Xorg :0 -br -pn
|                ├─292 /usr/bin/Xorg :0 -br -pn
|                ├─353 matchbox-window-manager -theme Sato -use_desktop_mode decorated -use_cursor no
|                ├─358 dbus-launch --sh-syntax --exit-with-session
|                ├─359 /usr/bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session
|                ├─376 connman-applet
|                ├─386 /usr/lib/gconf/gconfd-2
|                ├─401 matchbox-desktop
|                ├─402 matchbox-panel --titlebar --start-applets showdesktop,windowselector --end-applets clock,battery,,systray,startup-notify,notify
|                └─403 /usr/bin/settings-daemon
| 
| ======================================================================
| FAIL: test_systemd_status (oeqa.runtime.systemd.SystemdServiceTests)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
|     return f(*args, **kwargs)
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/runtime/systemd.py", line 77, in test_systemd_status
|     self.systemctl('status --full', 'avahi-daemon.service')
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/runtime/systemd.py", line 21, in systemctl
|     self.assertEqual(status, expected, message)
| AssertionError: 3 != 0 : systemctl status --full avahi-daemon.service
| ● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
|    Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
|    Active: failed (Result: timeout) since Fri 2016-01-15 10:58:55 UTC; 19min ago
|  Main PID: 262 (code=exited, status=0/SUCCESS)
|    Status: "avahi-daemon 0.6.31 starting up."
| 
| Jan 15 10:58:23 qemux86-64 avahi-daemon[262]: Server startup complete. Host name is qemux86-64.local. Local service cookie is 3756040364.
| Jan 15 10:58:24 qemux86-64 avahi-daemon[262]: Service "qemux86-64" (/services/ssh.service) successfully established.
| Jan 15 10:58:24 qemux86-64 avahi-daemon[262]: Service "qemux86-64" (/services/sftp-ssh.service) successfully established.
| Jan 15 10:58:55 qemux86-64 systemd[1]: avahi-daemon.service: Start operation timed out. Terminating.
| Jan 15 10:58:55 qemux86-64 avahi-daemon[262]: Got SIGTERM, quitting.
| Jan 15 10:58:55 qemux86-64 avahi-daemon[262]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.7.6.
| Jan 15 10:58:55 qemux86-64 avahi-daemon[262]: avahi-daemon 0.6.31 exiting.
| Jan 15 10:58:55 qemux86-64 systemd[1]: Failed to start Avahi mDNS/DNS-SD Stack.
| Jan 15 10:58:55 qemux86-64 systemd[1]: avahi-daemon.service: Unit entered failed state.
| Jan 15 10:58:55 qemux86-64 systemd[1]: avahi-daemon.service: Failed with result 'timeout'.
| 
| ----------------------------------------------------------------------
Comment 1 Randy MacLeod 2016-01-21 16:26:07 UTC 
This bug seems to be specific to the YP autobuilder.
Is the system just overloaded or dropping packets?

Yi, Have you seen this problem at all? Can you run an overnight test loop to help us understand how rare it is and if it can be reproduced on non-YP test instrastructure. Thanks.
Comment 2 Yi Zhao 2016-01-22 07:04:26 UTC 
(In reply to comment #1)
> This bug seems to be specific to the YP autobuilder.
> Is the system just overloaded or dropping packets?
> 
> Yi, Have you seen this problem at all? Can you run an overnight test loop to
> help us understand how rare it is and if it can be reproduced on non-YP test
> instrastructure. Thanks.

I hadn't seen this before since our local autobuilder doesn't build systemd image. I will try do reproduce it in local.
Comment 3 Paul Eggleton 2016-04-07 22:22:12 UTC 
This bug looks very similar to bug 8142 - assigning this one to Anibal as well.
Comment 4 Aníbal Limón 2016-04-08 17:31:03 UTC 
For me this bug is the same as 8142 so closed as duplicated.

The main reason for this is because systemd is killing avahi, blutooth, wpa_supplicant and ofono due to timeout.

Example:

| Jan 15 10:58:55 qemux86-64 systemd[1]: avahi-daemon.service: Start operation timed out. Terminating.
| Jan 15 10:58:55 qemux86-64 avahi-daemon[262]: Got SIGTERM, quitting.
| Jan 15 10:58:55 qemux86-64 avahi-daemon[262]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.7.6.

*** This bug has been marked as a duplicate of bug 8142 ***