| Summary: | runtime tests sometimes can't login to the serial console on systemd images (generates warning) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Richard Purdie <richard.purdie> | ||||||
| Component: | oeqa/runtime | Assignee: | Richard Purdie <richard.purdie> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | Medium+ | CC: | akuster808, akuster, alex.kanavin, JPEWhacker, raj.khem, randy.macleod, sgw, steve, tim.orling, tvgamblin | ||||||
| Version: | unspecified | ||||||||
| Target Milestone: | 3.2 M4 | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | AB-INT | ||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||
| Attachments: |
|
||||||||
|
Description
Richard Purdie
2019-11-21 23:00:13 UTC
Kai, Can you try to reproduce this defect? It seems that we (WR) really need to set-up an autobuildler instance. I wonder if anyone at the Beijing site has done that... could this be related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=13603 Can't reproduce the test image warning locally following the auto.conf. I'll do more work to figure out what is wrong. Trevor have you ever seen this in your autobuilder instance? Could you try to make it happen? (In reply to comment #8) > Trevor have you ever seen this in your autobuilder instance? Could you try > to make it happen? It happens most frequently with qemumips, perhaps as its slowest. No, I haven't seen this, but I'll keep an eye out for it now that we're setting an autobuilder up for testing. same warning from qemuarm with systemd https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1086 Its possible the latest attempt at fixing 13950 may also influence this (change the janitor nice level) I met some errors with commit [60d3f65d1c kernel-devsrc: fix on-target module build for v5.8+]. But it seems not the same as in auto builder. Follow the steps in https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1086 and run command continuously bitbake core-image-full-cmdline:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage -k Error messages: runqemu - ERROR - Failed to run qemu: qemu-system-arm: could not configure /dev/net/tun (tap1): Device or resource busy It seems qemu failed to start caused by lack of tap device. But in autobuilder, it is just warning that can't log in. What is interesting from that warning on the autobuilder is: WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into serial console as root using blank password WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output: root [ 302.102010] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 302.108301] NFSD: Using legacy client tracking operations. [ 302.134499] NFSD: starting 90-second grace period (net f0000039) [ 342.181837] rcu: INFO: rcu_preempt self-detected stall on CPU [ 342.183255] rcu: 0-...!: (1 ticks this GP) idle=37e/1/0x40000002 softirq=9401/9401 fqs=0 [ 342.184048] (t=2276 jiffies g=17905 q=6) [ 342.184453] rcu: rcu_preempt kthread starved for 2276 jiffies! g17905 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0 [ 342.185176] rcu: RCU grace-period kthread stack dump: [ 342.185612] rcu_preempt I 0 10 2 0x00000000 [ 342.186183] [<c09c5170>] (__schedule) from [<c09c52e8>] (schedule+0xbc/0xf4) [ 342.186755] [<c09c52e8>] (schedule) from [<c09c793c>] (schedule_timeout+0xcc/0xfc) [ 342.187343] [<c09c793c>] (schedule_timeout) from [<c0284d4c>] (rcu_gp_kthread+0x618/0x8ac) [ 342.187967] [<c0284d4c>] (rcu_gp_kthread) from [<c024d150>] (kthread+0x124/0x130) [ 342.188531] [<c024d150>] (kthread) from [<c02010d8>] (ret_from_fork+0x14/0x3c) [ 342.189071] Exception stack(0xdb50bfb0 to 0xdb50bff8) [ 342.189512] bfa0: 00000000 00000000 00000000 00000000 [ 342.190123] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 342.190782] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 342.191630] NMI backtrace for cpu 0 [ 342.192239] CPU: 0 PID: 258 Comm: sync Not tainted 5.4.50-yocto-standard #1 [ 342.192772] Hardware name: Generic DT based system [ 342.193267] [<c02113f8>] (unwind_backtrace) from [<c020bde4>] (show_stack+0x18/0x1c) [ 342.193880] [<c020bde4>] (show_stack) from [<c09af488>] (dump_stack+0x84/0xa0) [ 342.194420] [<c09af488>] (dump_stack) from [<c09b56c0>] (nmi_cpu_backtrace+0xa0/0xb0) [ 342.194988] [<c09b56c0>] (nmi_cpu_backtrace) from [<c09b578c>] (nmi_trigger_cpumask_backtrace+0xbc/0x15c) [ 342.195677] [<c09b578c>] (nmi_trigger_cpumask_backtrace) from [<c0287bf0>] (rcu_dump_cpu_stacks+0xac/0xc8) [ 342.196373] [<c0287bf0>] (rcu_dump_cpu_stacks) from [<c0286c94>] (rcu_sched_clock_irq+0x310/0x89c) [ 342.197108] [<c0286c94>] (rcu_sched_clock_irq) from [<c028ee4c>] (update_process_times+0x34/0x50) [ 342.197767] [<c028ee4c>] (update_process_times) from [<c029fc40>] (tick_sched_timer+0x15c/0x1c4) [ 342.198424] [<c029fc40>] (tick_sched_timer) from [<c028fc40>] (__hrtimer_run_queues+0x180/0x2d4) [ 342.199075] [<c028fc40>] (__hrtimer_run_queues) from [<c0290450>] (hrtimer_interrupt+0xc8/0x208) [ 342.199733] [<c0290450>] (hrtimer_interrupt) from [<c082bc90>] (arch_timer_handler_virt+0x30/0x38) [ 342.200381] [<c082bc90>] (arch_timer_handler_virt) from [<c027ad24>] (handle_percpu_devid_irq+0x60/0x118) [ 342.201071] [<c027ad24>] (handle_percpu_devid_irq) from [<c0275180>] (generic_handle_irq+0x20/0x30) [ 342.201742] [<c0275180>] (generic_handle_irq) from [<c02757a8>] (__handle_domain_irq+0x98/0xa8) [ 342.202391] [<c02757a8>] (__handle_domain_irq) from [<c0676784>] (gic_handle_irq+0x5c/0x94) [ 342.203020] [<c0676784>] (gic_handle_irq) from [<c0202504>] (__irq_usr+0x44/0x60) [ 342.203568] Exception stack(0xd6a35fb0 to 0xd6a35ff8) [ 342.203981] 5fa0: b6fbbf38 00000804 fbad2086 b6fbbf38 [ 342.204566] 5fc0: b6fbbf38 fbad2086 00000000 b6fbe8d0 b6fbe8d0 00000000 00000002 b6fbb2bc [ 342.205164] 5fe0: 004a0f4c beecfc78 0048f741 0048f790 20030030 ffffffff <<< run_serial(): command timed out after 60 seconds without output >>> which is a kernel backtrace about slow timers. It suggests the system was under considerable load and running slowly. The tap device error is normal for runqemu, it will print a message like that for each device it can't use but then use one when it finds on available. That would only be fatal if it ran out of devices (the autobuilders have 32 of them if I remember correctly). https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/818 fedora31-ty-1 core-image-sato-sdk qemumips-alt (systemd) https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/887 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1299 https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/880 while working on other qmeumips-alt issues, I found that one of the three testimages would not communicate over serial. I increased the mem to 512 and that issue went away. The systemd errors indicated not enough memory. There is a patch done by Ross to decrease memory do to instability. Thanks a lot @akuster. If you have a patch already, would you like to take this bug please? Since we're struggling to get anywhere with this, I did some testing. The main problem always seems to be the systemd based images and it tends to be core-image-sato-sdk. I added a bit of debug and on my local build system I saw: WARNING: core-image-minimal-1.0-r0 do_testimage: Took 8.258012294769287 for serial login WARNING: core-image-sato-1.0-r0 do_testimage: Took 11.680735111236572 for serial login WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Took 54.227009296417236 for serial login so on an unloaded system we see sato-sdk is very close to the 60s timeout. I'd love to understand why sato-sdk systemd is so slow to respond to an entry to a login prompt but perhaps we need to increase the timeout for now? I personally think the memory issue is a red herring FWIW. Patch to enable that logging (but not right as run_serial can be used for things other than login).
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 519aa9aa1e5..d2f9f601bcf 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -520,6 +521,7 @@ class QemuRunner:
data += answer.decode('utf-8')
# Search the prompt to stop
if re.search(self.boot_patterns['search_cmd_finished'], data):
+ self.logger.warning("Took %s for serial login" % str(now - start))
break
else:
raise Exception("No data on serial console socket")
Kai, Saul has started to work on this bug and he sees a delay in qemumips* due to ssh key generation. That might be the root cause but it's too early to say. He'll provide more info at some point, and hopefully it will include an explanation of how he debugged the issue so that we can learn from his experiences. Some very preliminary digging a little deeper using the systemd-analyze tool. I saw that one difference was sato uses dropbear and sato-sdk uses openssh, this means there sshdkeygen that takes some time. Setting for systemd, testimage: MACHINE = "qemumips" DISTRO_FEATURES_append = " systemd" DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit" VIRTUAL-RUNTIME_login_manager = "shadow-base" VIRTUAL-RUNTIME_init_manager = "systemd" VIRTUAL-RUNTIME_initscripts = "systemd-compat-units" INHERIT += "testimage" IMAGE_INSTALL += "systemd-analyze" I made the change Richard suggested above and ran bitbake -c testimage core-image-minimal ... on my server machine, the numbers were not as dramatic: core-image-minimal-1.0-r0 do_testimage: Took 3.3077690601348877 for serial login core-image-sato-1.0-r0 do_testimage: Took 2.004570245742798 for serial login core-image-sato-sdk-1.0-r0 do_testimage: Took 12.070182800292969 for serial login Using systemd-analyze blame and systemd-analyze critical-chain, I saw that the keygeneration took some time and I also saw something to do with ldconfig (ld.so.cache generation). More tomorrow. (In reply to comment #26) > Some very preliminary digging a little deeper using the systemd-analyze > tool. I saw that one difference was sato uses dropbear and sato-sdk uses > openssh, this means there sshdkeygen that takes some time. Both should need to generate keys but that is a significant difference between the two images types. I should perhaps switch to dropbear in my core-image-sato-sdk, see if the timings improve. I did double check there is rng pass through and there is. > Setting for systemd, testimage: > MACHINE = "qemumips" > > DISTRO_FEATURES_append = " systemd" > DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit" > VIRTUAL-RUNTIME_login_manager = "shadow-base" > VIRTUAL-RUNTIME_init_manager = "systemd" > VIRTUAL-RUNTIME_initscripts = "systemd-compat-units" > > INHERIT += "testimage" > > IMAGE_INSTALL += "systemd-analyze" > > I made the change Richard suggested above and ran bitbake -c testimage > core-image-minimal ... on my server machine, the numbers were not as > dramatic: > > core-image-minimal-1.0-r0 do_testimage: Took 3.3077690601348877 for serial > login > core-image-sato-1.0-r0 do_testimage: Took 2.004570245742798 for serial login > core-image-sato-sdk-1.0-r0 do_testimage: Took 12.070182800292969 for serial > login > > Using systemd-analyze blame and systemd-analyze critical-chain, I saw that > the keygeneration took some time and I also saw something to do with > ldconfig (ld.so.cache generation). > > More tomorrow. I'm puzzled as to how your qemumips is running so much faster than mine! root@qemumips:~# systemd-analyze critical-chain serial-getty@ttyS0.service The time when unit became active or started is printed after the "@" character. The time the unit took to start is printed after the "+" character. serial-getty@ttyS0.service @48.060s └─systemd-user-sessions.service @43.714s +840ms └─network.target @43.564s └─wpa_supplicant.service @40.501s +1.363s └─basic.target @22.285s └─sockets.target @22.267s └─sshd.socket @21.698s +337ms └─sysinit.target @21.546s └─systemd-userdbd.service @56.604s +2.656s └─systemd-userdbd.socket @3.443s └─-.mount @1.395s └─system.slice @1.392s └─-.slice @1.393s Which explains that the serial getty depends on networking which depends on sshd. It probably shouldn't. Currently sockets.target depends on sshd.socket. I think sshd.socket should be required by multi-user.target instead, that is what my desktop system has. This may explain some slow systemd boots in general. The dropbear/openssh unit files have been untouched since 2014 so it seems surprisingly nobody else noticed this until now but its possible, I did some investigation, and it looks like there are a few things conspiring to cause the slow down.
The explanation as to why some have seen the slowness and other have not is that "pam" must be in DISTRO_FEATURES to see the slow down. Adding this distro feature has two effects:
1) It takes longer for serial-getty@ttyS0 service to start because that service is marked with "After=systemd-user-sessions.service", and this service is only present if systemd was compiled with pam support
2) Enabling pam support causes additional behaviors when logind starts a user session. At a minimum, we know from strace logs that it's spending a long time waiting on a D-Bus call, and also when the user logs in, systemd spins up a systemd user session to go with it. You can see this after logging in:
root@qemumips:~# systemctl --user status
* qemumips
State: running
Jobs: 0 queued
Failed: 0 units
Since: Thu 2020-09-17 18:29:37 UTC; 21min ago
CGroup: /user.slice/user-0.slice/user@0.service
`-init.scope
|-263 /lib/systemd/systemd --user
`-268 (sd-pam)
I suspect the two behaviors are linked (that is, the long D-Bus wait is systemd spinning up the user session).
Created attachment 4722 [details]
Boot chart of a slow boot capture with `systemd-analyze plot`
Created attachment 4723 [details]
Boot chart of a boot with deferred ssh key generation
This boot chart shows a boot where the ssh key generatation is deferred until the ssh daemon accepts a client, but otherwise the same as the previous one. It shows that the key generation is consuming a lot of CPU cycles and slowing down the boot.
Lets disable gshadow in systemd. Atleast for qemu machine perhaps. I have sent a patch for that to test. Its on ml. Adding pre-generated ssh keys to many images has helped reduce the boot time and boot load which helps solve this issue. We did also 'nice' the key generate at boot. bulk change to add Saul's non-WR address. Bulk change: Remove Saul's old WR address. |