Bug 13646

Summary: runtime tests sometimes can't login to the serial console on systemd images (generates warning)
Product: [QA/Testing] Runtime Testing Reporter: Richard Purdie <richard.purdie>
Component: oeqa/runtimeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: akuster808, akuster, alex.kanavin, JPEWhacker, raj.khem, randy.macleod, sgw, steve, tim.orling, tvgamblin
Version: unspecified   
Target Milestone: 3.2 M4   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Boot chart of a slow boot capture with `systemd-analyze plot`
none
Boot chart of a boot with deferred ssh key generation none

Description Richard Purdie 2019-11-21 23:00:13 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/89

step1c: WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into serial console as root using blank password
step1c: WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:

We've seen this multiple times now
Comment 1 Randy MacLeod 2019-11-21 23:05:56 UTC
Kai, Can you try to reproduce this defect? 
It seems that we (WR) really need to set-up an autobuildler instance.
I wonder if anyone at the Beijing site has done that...
Comment 2 Armin Kuster 2019-12-04 16:05:17 UTC
could this be related to 

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13603
Comment 3 Kai Kang 2019-12-09 01:45:08 UTC
Can't reproduce the test image warning locally following the auto.conf. I'll do more work to figure out what is wrong.
Comment 8 Randy MacLeod 2020-07-16 07:44:49 UTC
Trevor have you ever seen this in your autobuilder instance? Could you try to make it happen?
Comment 10 Richard Purdie 2020-07-16 11:17:46 UTC
(In reply to comment #8)
> Trevor have you ever seen this in your autobuilder instance? Could you try
> to make it happen?

It happens most frequently with qemumips, perhaps as its slowest.
Comment 11 Trevor Gamblin 2020-07-16 11:27:09 UTC
No, I haven't seen this, but I'll keep an eye out for it now that we're setting an autobuilder up for testing.
Comment 12 Richard Purdie 2020-07-18 03:03:10 UTC
same warning from qemuarm with systemd https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1086
Comment 13 Richard Purdie 2020-07-22 14:39:35 UTC
Its possible the latest attempt at fixing 13950 may also influence this (change the janitor nice level)
Comment 14 Kai Kang 2020-07-27 02:37:18 UTC
I met some errors with commit [60d3f65d1c kernel-devsrc: fix on-target module build for v5.8+]. But it seems not the same as in auto builder.

Follow the steps in https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/1086

and run command continuously
bitbake core-image-full-cmdline:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage -k

Error messages:
runqemu - ERROR - Failed to run qemu: qemu-system-arm: could not configure /dev/net/tun (tap1): Device or resource busy

It seems qemu failed to start caused by lack of tap device. But in autobuilder, it is just warning that can't log in.
Comment 15 Richard Purdie 2020-07-27 08:57:08 UTC
What is interesting from that warning on the autobuilder is:

WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into serial console as root using blank password
WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
root
[  302.102010] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[  302.108301] NFSD: Using legacy client tracking operations.
[  302.134499] NFSD: starting 90-second grace period (net f0000039)
[  342.181837] rcu: INFO: rcu_preempt self-detected stall on CPU
[  342.183255] rcu: 	0-...!: (1 ticks this GP) idle=37e/1/0x40000002 softirq=9401/9401 fqs=0 
[  342.184048] 	(t=2276 jiffies g=17905 q=6)
[  342.184453] rcu: rcu_preempt kthread starved for 2276 jiffies! g17905 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[  342.185176] rcu: RCU grace-period kthread stack dump:
[  342.185612] rcu_preempt     I    0    10      2 0x00000000
[  342.186183] [<c09c5170>] (__schedule) from [<c09c52e8>] (schedule+0xbc/0xf4)
[  342.186755] [<c09c52e8>] (schedule) from [<c09c793c>] (schedule_timeout+0xcc/0xfc)
[  342.187343] [<c09c793c>] (schedule_timeout) from [<c0284d4c>] (rcu_gp_kthread+0x618/0x8ac)
[  342.187967] [<c0284d4c>] (rcu_gp_kthread) from [<c024d150>] (kthread+0x124/0x130)
[  342.188531] [<c024d150>] (kthread) from [<c02010d8>] (ret_from_fork+0x14/0x3c)
[  342.189071] Exception stack(0xdb50bfb0 to 0xdb50bff8)
[  342.189512] bfa0:                                     00000000 00000000 00000000 00000000
[  342.190123] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  342.190782] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  342.191630] NMI backtrace for cpu 0
[  342.192239] CPU: 0 PID: 258 Comm: sync Not tainted 5.4.50-yocto-standard #1
[  342.192772] Hardware name: Generic DT based system
[  342.193267] [<c02113f8>] (unwind_backtrace) from [<c020bde4>] (show_stack+0x18/0x1c)
[  342.193880] [<c020bde4>] (show_stack) from [<c09af488>] (dump_stack+0x84/0xa0)
[  342.194420] [<c09af488>] (dump_stack) from [<c09b56c0>] (nmi_cpu_backtrace+0xa0/0xb0)
[  342.194988] [<c09b56c0>] (nmi_cpu_backtrace) from [<c09b578c>] (nmi_trigger_cpumask_backtrace+0xbc/0x15c)
[  342.195677] [<c09b578c>] (nmi_trigger_cpumask_backtrace) from [<c0287bf0>] (rcu_dump_cpu_stacks+0xac/0xc8)
[  342.196373] [<c0287bf0>] (rcu_dump_cpu_stacks) from [<c0286c94>] (rcu_sched_clock_irq+0x310/0x89c)
[  342.197108] [<c0286c94>] (rcu_sched_clock_irq) from [<c028ee4c>] (update_process_times+0x34/0x50)
[  342.197767] [<c028ee4c>] (update_process_times) from [<c029fc40>] (tick_sched_timer+0x15c/0x1c4)
[  342.198424] [<c029fc40>] (tick_sched_timer) from [<c028fc40>] (__hrtimer_run_queues+0x180/0x2d4)
[  342.199075] [<c028fc40>] (__hrtimer_run_queues) from [<c0290450>] (hrtimer_interrupt+0xc8/0x208)
[  342.199733] [<c0290450>] (hrtimer_interrupt) from [<c082bc90>] (arch_timer_handler_virt+0x30/0x38)
[  342.200381] [<c082bc90>] (arch_timer_handler_virt) from [<c027ad24>] (handle_percpu_devid_irq+0x60/0x118)
[  342.201071] [<c027ad24>] (handle_percpu_devid_irq) from [<c0275180>] (generic_handle_irq+0x20/0x30)
[  342.201742] [<c0275180>] (generic_handle_irq) from [<c02757a8>] (__handle_domain_irq+0x98/0xa8)
[  342.202391] [<c02757a8>] (__handle_domain_irq) from [<c0676784>] (gic_handle_irq+0x5c/0x94)
[  342.203020] [<c0676784>] (gic_handle_irq) from [<c0202504>] (__irq_usr+0x44/0x60)
[  342.203568] Exception stack(0xd6a35fb0 to 0xd6a35ff8)
[  342.203981] 5fa0:                                     b6fbbf38 00000804 fbad2086 b6fbbf38
[  342.204566] 5fc0: b6fbbf38 fbad2086 00000000 b6fbe8d0 b6fbe8d0 00000000 00000002 b6fbb2bc
[  342.205164] 5fe0: 004a0f4c beecfc78 0048f741 0048f790 20030030 ffffffff
<<< run_serial(): command timed out after 60 seconds without output >>>

which is a kernel backtrace about slow timers. It suggests the system was under considerable load and running slowly.

The tap device error is normal for runqemu, it will print a message like that for each device it can't use but then use one when it finds on available. That would only be fatal if it ran out of devices (the autobuilders have 32 of them if I remember correctly).
Comment 17 Richard Purdie 2020-08-16 15:06:33 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/818 fedora31-ty-1 core-image-sato-sdk qemumips-alt (systemd)
Comment 21 akuster 2020-09-04 10:28:14 UTC
while working on other qmeumips-alt issues, I found that one of the three testimages would not communicate over serial. 

I increased the mem to 512 and that issue went away. The systemd errors indicated not enough memory.

There is a patch done by Ross to decrease memory do to instability.
Comment 22 Kai Kang 2020-09-05 21:02:42 UTC
Thanks a lot @akuster. If you have a patch already, would you like to take this bug please?
Comment 23 Richard Purdie 2020-09-08 06:19:52 UTC
Since we're struggling to get anywhere with this, I did some testing. The main problem always seems to be the systemd based images and it tends to be core-image-sato-sdk.

I added a bit of debug and on my local build system I saw:

WARNING: core-image-minimal-1.0-r0 do_testimage: Took 8.258012294769287 for serial login
WARNING: core-image-sato-1.0-r0 do_testimage: Took 11.680735111236572 for serial login
WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Took 54.227009296417236 for serial login

so on an unloaded system we see sato-sdk is very close to the 60s timeout. I'd love to understand why sato-sdk systemd is so slow to respond to an entry to a login prompt but perhaps we need to increase the timeout for now?

I personally think the memory issue is a red herring FWIW.
Comment 24 Richard Purdie 2020-09-08 06:23:03 UTC
Patch to enable that logging (but not right as run_serial can be used for things other than login).

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 519aa9aa1e5..d2f9f601bcf 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -520,6 +521,7 @@ class QemuRunner:
                     data += answer.decode('utf-8')
                     # Search the prompt to stop
                     if re.search(self.boot_patterns['search_cmd_finished'], data):
+                        self.logger.warning("Took %s for serial login" % str(now - start))
                         break
                 else:
                     raise Exception("No data on serial console socket")
Comment 25 Randy MacLeod 2020-09-10 15:24:31 UTC
Kai, 
Saul has started to work on this bug and he sees a delay in qemumips* due to ssh key generation. That might be the root cause but it's too early to say. He'll provide more info at some point, and hopefully it will include an explanation of how he debugged the issue so that we can learn from his experiences.
Comment 26 Saul Wold 2020-09-10 17:52:14 UTC
Some very preliminary digging a little deeper using the systemd-analyze tool. I saw that one difference was sato uses dropbear and sato-sdk uses openssh, this means there sshdkeygen that takes some time.


Setting for systemd, testimage:
MACHINE = "qemumips"

DISTRO_FEATURES_append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
VIRTUAL-RUNTIME_login_manager = "shadow-base"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"

INHERIT += "testimage"

IMAGE_INSTALL += "systemd-analyze"

I made the change Richard suggested above and ran bitbake -c testimage core-image-minimal ... on my server machine, the numbers were not as dramatic:

core-image-minimal-1.0-r0 do_testimage: Took 3.3077690601348877 for serial login
core-image-sato-1.0-r0 do_testimage: Took 2.004570245742798 for serial login
core-image-sato-sdk-1.0-r0 do_testimage: Took 12.070182800292969 for serial login

Using systemd-analyze blame and systemd-analyze critical-chain, I saw that the keygeneration took some time and I also saw something to do with ldconfig (ld.so.cache generation).

More tomorrow.
Comment 27 Richard Purdie 2020-09-16 09:25:16 UTC
(In reply to comment #26)
> Some very preliminary digging a little deeper using the systemd-analyze
> tool. I saw that one difference was sato uses dropbear and sato-sdk uses
> openssh, this means there sshdkeygen that takes some time.

Both should need to generate keys but that is a significant difference between the two images types. I should perhaps switch to dropbear in my core-image-sato-sdk, see if the timings improve.

I did double check there is rng pass through and there is.

> Setting for systemd, testimage:
> MACHINE = "qemumips"
> 
> DISTRO_FEATURES_append = " systemd"
> DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
> VIRTUAL-RUNTIME_login_manager = "shadow-base"
> VIRTUAL-RUNTIME_init_manager = "systemd"
> VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"
> 
> INHERIT += "testimage"
> 
> IMAGE_INSTALL += "systemd-analyze"
> 
> I made the change Richard suggested above and ran bitbake -c testimage
> core-image-minimal ... on my server machine, the numbers were not as
> dramatic:
> 
> core-image-minimal-1.0-r0 do_testimage: Took 3.3077690601348877 for serial
> login
> core-image-sato-1.0-r0 do_testimage: Took 2.004570245742798 for serial login
> core-image-sato-sdk-1.0-r0 do_testimage: Took 12.070182800292969 for serial
> login
> 
> Using systemd-analyze blame and systemd-analyze critical-chain, I saw that
> the keygeneration took some time and I also saw something to do with
> ldconfig (ld.so.cache generation).
> 
> More tomorrow.

I'm puzzled as to how your qemumips is running so much faster than mine!
Comment 28 Richard Purdie 2020-09-17 06:50:27 UTC
root@qemumips:~# systemd-analyze critical-chain serial-getty@ttyS0.service
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

serial-getty@ttyS0.service @48.060s
└─systemd-user-sessions.service @43.714s +840ms
  └─network.target @43.564s
    └─wpa_supplicant.service @40.501s +1.363s
      └─basic.target @22.285s
        └─sockets.target @22.267s
          └─sshd.socket @21.698s +337ms
            └─sysinit.target @21.546s
              └─systemd-userdbd.service @56.604s +2.656s
                └─systemd-userdbd.socket @3.443s
                  └─-.mount @1.395s
                    └─system.slice @1.392s
                      └─-.slice @1.393s

Which explains that the serial getty depends on networking which depends on sshd. It probably shouldn't.
Comment 29 Richard Purdie 2020-09-17 07:10:01 UTC
Currently sockets.target depends on sshd.socket. I think sshd.socket should be required by multi-user.target instead, that is what my desktop system has. This may explain some slow systemd boots in general. The dropbear/openssh unit files have been untouched since 2014 so it seems surprisingly nobody else noticed this until now but its possible,
Comment 30 Joshua Watt 2020-09-17 11:54:25 UTC
I did some investigation, and it looks like there are a few things conspiring to cause the slow down.

The explanation as to why some have seen the slowness and other have not is that "pam" must be in DISTRO_FEATURES to see the slow down. Adding this distro feature has two effects:

 1) It takes longer for serial-getty@ttyS0 service to start because that service is marked with "After=systemd-user-sessions.service", and this service is only present if systemd was compiled with pam support

 2) Enabling pam support causes additional behaviors when logind starts a user session. At a minimum, we know from strace logs that it's spending a long time waiting on a D-Bus call, and also when the user logs in, systemd spins up a systemd user session to go with it. You can see this after logging in:

 root@qemumips:~# systemctl --user status
 * qemumips
     State: running
      Jobs: 0 queued
    Failed: 0 units
     Since: Thu 2020-09-17 18:29:37 UTC; 21min ago
    CGroup: /user.slice/user-0.slice/user@0.service
            `-init.scope 
              |-263 /lib/systemd/systemd --user
              `-268 (sd-pam)

I suspect the two behaviors are linked (that is, the long D-Bus wait is systemd spinning up the user session).
Comment 31 Joshua Watt 2020-09-17 11:56:25 UTC
Created attachment 4722 [details]
Boot chart of a slow boot capture with `systemd-analyze plot`
Comment 32 Joshua Watt 2020-09-17 13:29:22 UTC
Created attachment 4723 [details]
Boot chart of a boot with deferred ssh key generation

This boot chart shows a boot where the ssh key generatation is deferred until the ssh daemon accepts a client, but otherwise the same as the previous one. It shows that the key generation is consuming a lot of CPU cycles and slowing down the boot.
Comment 33 Khem Raj 2020-09-18 17:19:32 UTC
Lets disable gshadow in systemd. Atleast for qemu machine perhaps. I have sent a patch for that to test. Its on ml.
Comment 34 Richard Purdie 2020-10-08 13:57:16 UTC
Adding pre-generated ssh keys to many images has helped reduce the boot time and boot load which helps solve this issue. We did also 'nice' the key generate at boot.
Comment 35 Randy MacLeod 2025-02-14 16:46:31 UTC
bulk change to add Saul's non-WR address.
Comment 36 Randy MacLeod 2025-02-14 16:48:16 UTC
Bulk change: Remove Saul's old WR address.