11277 2017-03-31 12:45:10 +0000 CI: enable the use of KVM for running selftests 2017-04-17 17:59:02 +0000 1 1 6 Yocto Project Subprojects IoT Reference OS Kit intel-iot-refkit-tools unspecified x86 Multiple RESOLVED FIXED Medium normal 2.3 M4 1 patrick.ohly olev.kartau mikko.ylinen --- --- No (bug/feature does not impact docs) oldest_to_newest 71961 0 patrick.ohly 2017-03-31 12:45:10 +0000 In https://github.com/intel/intel-iot-refkit/pull/97, QEMU_USE_KVM couldn't be set because the CI environment doesn't seem to have kvm enabled. From http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html For KVM to work, all the following conditions must be met: Your MACHINE must be either qemux86" or "qemux86-64". Your build host has to have the KVM modules installed, which are /dev/kvm. The build host /dev/kvm directory has to be both writable and readable. Not having KVM makes qemu slower, in my experience by a factor of four or more. 72054 1 mikko.ylinen 2017-04-04 06:01:39 +0000 (In reply to comment #0) > > Your MACHINE must be either qemux86" or "qemux86-64". Do you know why's this a dependency? 72058 2 patrick.ohly 2017-04-04 06:44:38 +0000 (In reply to comment #1) > (In reply to comment #0) > > > > > Your MACHINE must be either qemux86" or "qemux86-64". > > Do you know why's this a dependency? I suspect the documentation is out-dated or over-simplifies the problem. Right now, those MACHINEs still work better under qemu (network configured correctly thanks to the ip boot parameter and some extra packages which apply that), but we recently also made it possible to run meta-intel MACHINEs. 72234 3 olev.kartau 2017-04-10 18:30:02 +0000 title is confusing. "enable the use of QEMU" is already satisfied as CI jobs have completed qemu sessions. Inside text of Description however there is KVM mentioned. KVM is one operating mode of QEMU. So what is actually asked is "enable KVM mode of QEMU"? Then about KVM. In current setup of pre- and post-build jobs, these all run in docker. Thats why it seems "no kvm support", although the hosts have KVM support. But it does not automatically propagate into docker session without adding some set-up for this, - we need to create device inside docker - we need to run docker in --privileged mode That's probably achievable, but then lets ask, is it necessary to pile two levels of virtualization for this to work? Can we run QEMU_KVM session in host, leaving docker out? This thread starts to touch another topic which came out when we looked how to run tests in parallel. And making tests running in parallel is harder if the pos-tests are technically implemented as build steps inside docker. From architecture POV, test steps should be independent from build steps, with possibility to run them on some other worker node, in parallel. But a post-test that runs in same docker session with builder, is not detachable to be parallel. 72248 4 patrick.ohly 2017-04-10 20:23:11 +0000 (In reply to comment #3) > So what is actually asked is "enable KVM mode of QEMU"? Yes. > Then about KVM. > In current setup of pre- and post-build jobs, > these all run in docker. > Thats why it seems "no kvm support", although the hosts have KVM support. > But it does not automatically propagate into docker session without > adding some set-up for this, > - we need to create device inside docker > - we need to run docker in --privileged mode > > That's probably achievable, but then lets ask, is it necessary > to pile two levels of virtualization for this to work? Selftests need the ability to invoke qemu. That's because they need to control how the virtual machine is configured, before booting it up. > Can we run QEMU_KVM session in host, leaving docker out? No, that won't work for the selftests. > This thread starts to touch another topic which came out when we looked > how to run tests in parallel. And making tests running in parallel is harder > if the pos-tests are technically implemented as build steps inside docker. > > From architecture POV, test steps should be independent from build steps, > with possibility to run them on some other worker node, in parallel. > But a post-test that runs in same docker session with builder, > is not detachable to be parallel. True in theory, but in practice I expect the performance to be better when sharing the same build directory. That's because the post-selftest benefit from the existing build artifacts, which are exactly where the tests needs them. Doing it on another node and in another environment is both more complicated (need to share at least the sstate) and slower (need to rebuild directories and images from sstate). Image testing is different because it has more-or-less well-defined input (but even there we have to hack around with non-upstreameable classes to make all required files available), whereas the selftest doesn't. 72283 5 olev.kartau 2017-04-11 11:33:40 +0000 currently these steps are performed in CI builder, in docker session: pre-build,build,store-images,post-build I think it's good idea to open privileges as little as possible, so can this work: 1. we run 3 steps in regular docker session, as present; 2. we add 2nd docker session in privileged mode, which can start qemu-kvm, where we will start 4th step: selftests Note that stopping of 1st docker session will lose runtime context of build session and keep everything which is in file system. Is this a problem? If above plan is not doable, we have to run the one docker session with wide privileges. 72284 6 patrick.ohly 2017-04-11 11:45:56 +0000 (In reply to comment #5) > currently these steps are performed in CI builder, in docker session: > pre-build,build,store-images,post-build > I think it's good idea to open privileges as little as possible, > so can this work: > 1. we run 3 steps in regular docker session, as present; > 2. we add 2nd docker session in privileged mode, which can start qemu-kvm, > where we will start 4th step: selftests > Note that stopping of 1st docker session will lose runtime context of build > session and keep everything which is in file system. Is this a problem? Shouldn't be a problem. However, I wonder why this is so hard. I don't know what --privileged does, so let me ask: is this required? Normally, /dev/kvm is group-owned by kvm. When the user starting docker is in that group, what special privileges does docker need? It doesn't need to grant any additional privileges. Granting the build job access to KVM doesn't worry me. 72285 7 olev.kartau 2017-04-11 12:05:18 +0000 I may have been reading some older examples (about --privileged mode required). Seems it is possible to pass rights to one device only (--device option). I will try is this enough to get kvm mode going. 72385 8 olev.kartau 2017-04-12 15:15:19 +0000 refkit PR#111 adds passing of /dev/kvm to docker session, and I verified with qemu calling code added (which I temporarily borrowed from PR#94), that -enable-kvm is supported. In addition to that, builder hosts have /dev/kvm set as group+rw and belong to jenkins group, i.e. the device is usable with same credentials as are running the processes in docker session. 72540 9 olev.kartau 2017-04-17 17:59:02 +0000 This was implemented in PR111, but that small commit was then also added to PR#97 which has now been merged, thus marking this as RESOLVED