Bug 12508 - qemu hangs while running oe-selftest
Summary: qemu hangs while running oe-selftest
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: virtual machines (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.5 M3
Assignee: Anuj Mittal
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-01-24 09:54 UTC by Anuj Mittal
Modified: 2018-02-14 01:31 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
qemurunner, failing case (4.03 KB, application/octet-stream)
2018-01-24 09:54 UTC, Anuj Mittal
no flags Details
qemu boot log, failing case (21.75 KB, application/octet-stream)
2018-01-24 09:55 UTC, Anuj Mittal
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Anuj Mittal 2018-01-24 09:54:40 UTC
Created attachment 4184 [details]
qemurunner, failing case

Observed when running:

oe-selftest -r imagefeatures.ImageFeatures.test_non_root_user_can_connect_via_ssh_without_password

This results in a hang of guest VM (this is a nested VM). It works if I remove -enable-kvm.

Happens on host kernels >= 4.14. Tried on F26 4.14.4, 4.14.14, 4.14.15, 4.15.0-rc8.

qemu also doesn't hang if invoked using: runqemu kvm nographic. However, it doesn't shut down cleanly & either hangs while shutting down or results in a defunct process.

[ab@fedora26 log]$ ps -aef | grep qemu
ab        2069     1  8 03:23 ?        00:04:10 [qemu-system-x86] <defunct>


This build has the changes from bug 12301 that is x2apic is enabled in kernel. I have attached the qemurunner and boot log. Even though it doesn't show it in the log, but qemu did boot up successfully and qemurunner reached the stage where it had detected the login banner. It froze somewhere after that.
Comment 1 Anuj Mittal 2018-01-24 09:55:08 UTC
Created attachment 4185 [details]
qemu boot log, failing case
Comment 2 Anuj Mittal 2018-02-02 07:21:20 UTC
Definitely an issue with nested KVM. Happens when a SIGTERM is sent to VM started by oe-selftest using kvm. I can't reproduce this when running on a physical machine.

A workaround could be to check cpuinfo flags in runqemu (check for hypervisor flag) to see if we're running a nested KVM and not use -enable-kvm.
Comment 3 Joshua Lock 2018-02-02 10:39:38 UTC
(In reply to comment #2)
> Definitely an issue with nested KVM. Happens when a SIGTERM is sent to VM
> started by oe-selftest using kvm. I can't reproduce this when running on a
> physical machine.

Good progress. This sounds like it would be enough information already to start filing a bug with upstream and get some of the KVM maintainers to look at the issue?

> A workaround could be to check cpuinfo flags in runqemu (check for
> hypervisor flag) to see if we're running a nested KVM and not use
> -enable-kvm.

That would work certainly around the crash issue. Unfortunately that will cost us in performance and thus QA execution time. In the longer term we really need to have this fixed upstream.
Comment 4 Anuj Mittal 2018-02-12 03:26:54 UTC
This is no longer reproducible after upgrading the L0 host to 4.14.
Comment 5 Anuj Mittal 2018-02-14 01:31:46 UTC
Upgrading hosts to 4.14 works.