Bug 12508

Summary: qemu hangs while running oe-selftest
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Anuj Mittal <anuj.mittal>
Component: virtual machinesAssignee: Anuj Mittal <anuj.mittal>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: aaron.chun.yew.chan, anuj.mittal, joshuagloe, juro.bystricky, randy.macleod
Version: unspecified   
Target Milestone: 2.5 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
qemurunner, failing case
none
qemu boot log, failing case none

Description Anuj Mittal 2018-01-24 09:54:40 UTC 
Created attachment 4184 [details]
qemurunner, failing case

Observed when running:

oe-selftest -r imagefeatures.ImageFeatures.test_non_root_user_can_connect_via_ssh_without_password

This results in a hang of guest VM (this is a nested VM). It works if I remove -enable-kvm.

Happens on host kernels >= 4.14. Tried on F26 4.14.4, 4.14.14, 4.14.15, 4.15.0-rc8.

qemu also doesn't hang if invoked using: runqemu kvm nographic. However, it doesn't shut down cleanly & either hangs while shutting down or results in a defunct process.

[ab@fedora26 log]$ ps -aef | grep qemu
ab        2069     1  8 03:23 ?        00:04:10 [qemu-system-x86] <defunct>


This build has the changes from bug 12301 that is x2apic is enabled in kernel. I have attached the qemurunner and boot log. Even though it doesn't show it in the log, but qemu did boot up successfully and qemurunner reached the stage where it had detected the login banner. It froze somewhere after that.
Comment 1 Anuj Mittal 2018-01-24 09:55:08 UTC 
Created attachment 4185 [details]
qemu boot log, failing case
Comment 2 Anuj Mittal 2018-02-02 07:21:20 UTC 
Definitely an issue with nested KVM. Happens when a SIGTERM is sent to VM started by oe-selftest using kvm. I can't reproduce this when running on a physical machine.

A workaround could be to check cpuinfo flags in runqemu (check for hypervisor flag) to see if we're running a nested KVM and not use -enable-kvm.
Comment 3 Joshua Lock 2018-02-02 10:39:38 UTC 
(In reply to comment #2)
> Definitely an issue with nested KVM. Happens when a SIGTERM is sent to VM
> started by oe-selftest using kvm. I can't reproduce this when running on a
> physical machine.

Good progress. This sounds like it would be enough information already to start filing a bug with upstream and get some of the KVM maintainers to look at the issue?

> A workaround could be to check cpuinfo flags in runqemu (check for
> hypervisor flag) to see if we're running a nested KVM and not use
> -enable-kvm.

That would work certainly around the crash issue. Unfortunately that will cost us in performance and thus QA execution time. In the longer term we really need to have this fixed upstream.
Comment 4 Anuj Mittal 2018-02-12 03:26:54 UTC 
This is no longer reproducible after upgrading the L0 host to 4.14.
Comment 5 Anuj Mittal 2018-02-14 01:31:46 UTC 
Upgrading hosts to 4.14 works.