Bug 5570

Summary:	the system hang in yocto when it resumes from s3 standby on minnow board
Product:	[Build System, Metadata & Runtime] BSPs	Reporter:	Qian Yi <yi.qian>
Component:	bsps-meta-minnow	Assignee:	Darren Hart <dvhart>
Status:	RESOLVED WONTFIX	QA Contact:
Severity:	normal
Priority:	Medium	CC:	scott.a.garman, sjolley.yp.pm, tom.zanussi, warthog9
Version:	1.5
Target Milestone:	1.8
Hardware:	MinnowBoard
OS:	x86
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Qian Yi 2013-11-20 06:17:28 UTC

I tried the master version (v1.5) of yocto image on minnow. As currently the power button can’t wake up the system from s3, I used the RTC to wake up the system. I typed the below commands in yocto console to enter s3 standby:

echo "+00-00-00 00:00:10">/proc/acpi/alarm
echo mem>/sys/power/state

In around ten seconds, the system was automatically woken up. And the bios successfully restored the memory and register values for those pci devices. Finally it transferred the control to the OS wake vector. After a while, the yocto had no response, no message printed and no display. Then I used the itp to halt, but it showed “[P1]:unable to halt target processor”.

To figure out the reason, i built the source code of yocto for minnow and installed the generated image. After debugging with itp, i found it hung at this function in linux kernel "enable_nonboot_cpus". Very occasionally, after long time of running, it can bypass this function and the system can resume to the yocto console successfully. But in most time, it can not go forward.

Comment 1 Darren Hart 2013-11-20 18:37:12 UTC

Qian Yi, thanks for the detailed report. Since you have the ITP setup and have made such progress already, is this something you can continue to look into? Do you have any indication if this is a firmware (e.g. ACPI?) or an OS issue? Can you tell on which line of enable_nonboot_cpus the system is hanging?

Does booting with the kernel parameter "maxcpus=1" or "nosmp" have any impact on resume?

Comment 2 Darren Hart 2013-11-21 15:48:48 UTC

Quian, we see a lot of issues like this with Queensbay platforms. Firmware is looking into issues at that level. I'm without an ITP, so if you can help here, that would be very helpful. I understand the severity setting of critical, but given the frequency of this on Queensbay, I'm dropping the severity to normal.

Comment 3 Qian Yi 2013-12-06 10:23:33 UTC

I have tried the boot option "nosmp". The yocto will boot with single cpu enabled. After the system is woken up from s3 sleep, the yocto will stop at the function "mwait_idle". The serial console and display are not recovered.

Then i still debugged with two cpus enabled. After several rounds of tracing the code, i found even closer the place the system hung at: in arch\x86\kernel\SmpBoot.c, function do_boot_cpu, line 813-824. It looks like the AP is not woken up initially and shows no response. The kernel will keep checking the cpu status until 5 seconds are timed out. If this long loop is bypassed, the s3 resume can be successful. Both the serial console and display are restore. It is very strange although the ap is not successfully brought up at the beginning, but can be restored later. See the log printed below.

Also i tried skipping the _cpu_up function's execution with itp, the s3 resume can also be successful.


[  156.973418] pch_gpio 0000:02:00.2: pci_enable_wake Failed --22
[  157.287166] PM: suspend of devices complete after 460.568 msecs
[  157.287914] PM: late suspend of devices complete after 0.733 msecs
[  157.288817] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.299168] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.310144] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.321142] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.332208] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.343145] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.354142] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.365155] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.376144] pcieport 0000:00:17.0: System wakeup enabled by ACPI
[  157.387375] PM: noirq suspend of devices complete after 99.449 msecs
[  157.387447] ACPI: Preparing to enter system sleep state S3
[  157.387566] PM: Saving platform NVS memory
[  157.409244] Disabling non-boot CPUs ...
[  157.411069] smpboot: CPU 1 is now offline
[  157.411842] ACPI: Low-level resume complete
[  157.411842] PM: Restoring platform NVS memory
[  248.883821] Enabling non-boot CPUs ...
[  289.234053] smpboot: Booting Node 0 Processor 1 APIC 0x1
[  294.418610] smpboot: CPU1: Not responding
[  327.816191] Error taking CPU1 up: -5
[  157.410023] Initializing CPU#1
[  421.790187] ACPI: Waking up from system sleep state S3
[  421.845146] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.856139] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.867137] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.878137] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.922137] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.933138] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.944137] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  421.955137] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  422.043544] PM: noirq resume of devices complete after 253.137 msecs
[  422.044426] PM: early resume of devices complete after 0.801 msecs
[  422.044936] pcieport 0000:00:17.0: System wakeup disabled by ACPI
[  422.045027] pch_gbe: pch-gbe.miim won't go Ready
[  422.048232] pch_gbe: pch-gbe.miim won't go Ready
[  422.122042] usb usb3: root hub lost power or was reset
[  422.123091] usb usb4: root hub lost power or was reset
[  422.124127] usb usb6: root hub lost power or was reset
[  422.125162] usb usb8: root hub lost power or was reset
[  422.126197] usb usb7: root hub lost power or was reset
[  422.127232] usb usb5: root hub lost power or was reset
[  422.127420] usb usb2: root hub lost power or was reset
[  422.127632] usb usb1: root hub lost power or was reset
[  422.359123] ata2: SATA link down (SStatus 0 SControl 300)
[  422.359261] ata1: SATA link down (SStatus 0 SControl 300)
[  422.397545] PM: resume of devices complete after 353.101 msecs
[  422.521294] Restarting tasks ... [  422.522621] journal_bmap: journal block not found at offset 271 on mmcblk0p2

Comment 4 Qian Yi 2014-01-09 07:04:40 UTC

I did many investigations like deadloop before mwait idle or not enable local apic timer in start_secondary function. Then i found the root cause of the resume hang is the races between local apic timer and deep c states on the secondary cpu especially in the resume path. To prove it, i tried the below boot options to disable local apic, or disable secondary cpu, or disable lower c states, they all can solve this issue:"nolapic_timer", "maxcpus=1", "intel_idle.max_cstates=1", "intel_idle.max_cstates=0 processor.max_cstates=0", . But for these boot options like "intel_idle.max_cstates=2", "intel_idle.max_cstates=4","intel_idle.max_cstates=0 processor.max_cstates=2", "intel_idle.max_cstates=0 processor.max_cstates=4", the issue is still there. from these facts, it is very obvious that the issue is caused by the conflicts between local apic timer and lower c states just as i concluded.

To fix it, i searched in internet and found it is well known issue in linux kernel. You can find some existent description about this issue in the file header of intel_idle.c, like this "ACPI has a .suspend hack to turn off deep c-statees during suspend to avoid complications with the lapic timer workaround. Have not seen issues with suspend, but may need same workaround here.". I collected some hot people's discussions like Rafael J. Wysocki who is a major contributor of linux kernel power management code. They provided some solutions, although these fixings are still not checked in in kernel, I think it will be very helpful for you to fix it in yocto.
http://marc.info/?t=134087458700003&r=1&w=2

Also it is said linaro might have solved this issue.

Comment 5 Stephen K Jolley 2014-05-29 15:21:12 UTC

Review the comments

Comment 6 John 'Warthog9' Hawley 2015-01-26 22:25:58 UTC

MinnowBoard v1 has reached new feature EOL, FW development is done as are new hardware features.  If you feel EXCEEDINGLY strongly about an issue please re-open it for discussion but be aware that we are recommending individuals migrate to the MAX platform.