I'm filing this because oeqa.selftest.runtime-test.Postinst (test_postinst_roofs_and_boot) seems to take very long to execute on the autobuilder . I'm not a subject matter expert so if the runtime is actually reasonable, please close. On an example run (March 3rd, https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/200): the test took 44 minutes(!), longer than any other test. This is 16% of the whole selftest time. Assigning to Jose based on git log.
(In reply to comment #0) > I'm filing this because oeqa.selftest.runtime-test.Postinst > (test_postinst_roofs_and_boot) seems to take very long to execute on the > autobuilder . I'm not a subject matter expert so if the runtime is actually > reasonable, please close. > > On an example run (March 3rd, > https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/200): the > test took 44 minutes(!), longer than any other test. This is 16% of the > whole selftest time. > > Assigning to Jose based on git log. Jussi The test is taking too long because is testing several combinations and also booting the image for each one of the combinations, the test was created based on Bug 8433 requirements that states as follows: --- "Build the image, check the contents *before* booting, verify that test-postinst-rootfs has ran it's maintainer scripts and -delayed has not. Boot in qemu, then verify that both -rootfs and -delayed have touched their files. Repeat for both sysvinit and systemd, and for rpm/ipk/dpkg (six combinations in total)." ---
Yeah I've noticed the mad test matrix, it doesn't make things easy. I still think we need to investigate: we should probably be testing much more things like this but if a single test adds 44 minutes... there's no way we can add these other tests without negatively affecting the autobuilder test cycle. Some statistics might be useful -- where is the time really spent? Then some quick ideas: you (or Ross for the AB question) can probably tell right away if they're stupid: * Would it help if only one package class is defined at a time? Could be it actually makes things worse... * Are autobuilders typically fully loaded CPU-wise? Can we do multiple runqemus at the same time? * you use "-c cleanall": Is that ok here? I don't quite understand where rootfs_pkg and boot_pkg actually come from... * There's a lot of appending going on with features (I think in the end PACKAGE_CLASSES is defined six times). I don't think this leads to anything nasty but might be worth checking...
With regards to statistics: I've filed https://bugzilla.yoctoproject.org/show_bug.cgi?id=11239 for DISTRO_FEATURE changes affecting native recipes: it might be interesting to know if there is any native recipe building going on in those repeating core-image-minimal builds...
(In reply to comment #2) > * you use "-c cleanall": Is that ok here? I don't quite understand where > rootfs_pkg and boot_pkg actually come from... It's probably not a major issue in this instance but: Joshua has a fix for this in a patch on mailing list.
WRT stats: The actual testing wihtout package builds (so setup, creating 6 images, booting 6 images, testing, teardown) takes 10-12 minutes for the six test cases altogether. This is not horrible but I bet not there's low-hanging fruit to pick there as well. So the bulk of the time is _probably_ spent configuring/building/packaging recipes. I can't actually confirm this properly as the way oe-selftest uses qemurunner seems to be broken (see bug 11254): I've been trying for three days but have been unable to run the complete oeqa.selftest.runtime-test.Postinst succesfully yet :( So some ideas with that assumption: * we should be able to keep "systemd" in DISTRO_FEATURES from the start, right? changing that midway means we end up re-building _a lot_ of stuff. * the test sets MACHINE = qemux86 but other tests in this file do not seem to (test_verify_postinst claims to in a comment but does not really seem to do that) -- are we sure we don't build lots of different archs for no reason?
Unfortunately there is no room for improving performance due to the nature of the test: test runs every combination of init system and package class and for the image create it, boot and test post install files. Resolving it as resolve/not-a-bug
(In reply to comment #5) > WRT stats: Sorry I resolved this bug without answering your questions first. > > The actual testing wihtout package builds (so setup, creating 6 images, > booting 6 images, testing, teardown) takes 10-12 minutes for the six test > cases altogether. This is not horrible but I bet not there's low-hanging > fruit to pick there as well. > > So the bulk of the time is _probably_ spent configuring/building/packaging > recipes. I can't actually confirm this properly as the way oe-selftest uses > qemurunner seems to be broken (see bug 11254): I've been trying for three > days but have been unable to run the complete > oeqa.selftest.runtime-test.Postinst succesfully yet :( > > So some ideas with that assumption: > * we should be able to keep "systemd" in DISTRO_FEATURES from the start, > right? changing that midway means we end up re-building _a lot_ of stuff. there are two loops, the outer one start with sysvinit then systemd, so system is just rebuilding stuff just once. > * the test sets MACHINE = qemux86 but other tests in this file do not seem > to (test_verify_postinst claims to in a comment but does not really seem to > do that) -- are we sure we don't build lots of different archs for no reason? good point. I have removed that line (so it is picking the default MACHINE which is qemux86) and observed that times remain the same. I will send a patch to remove that line.
(In reply to comment #7) > > So some ideas with that assumption: > > * we should be able to keep "systemd" in DISTRO_FEATURES from the start, > > right? changing that midway means we end up re-building _a lot_ of stuff. > > there are two loops, the outer one start with sysvinit then systemd, so > system is just rebuilding stuff just once. I see that, I'm trying to avoid those package rebuilds even if it only happens once. My question was: is there a benefit to not having "systemd" in DISTRO_FEATURES all the time?
(In reply to comment #8) > (In reply to comment #7) > > > So some ideas with that assumption: > > > * we should be able to keep "systemd" in DISTRO_FEATURES from the start, > > > right? changing that midway means we end up re-building _a lot_ of stuff. > > > > there are two loops, the outer one start with sysvinit then systemd, so > > system is just rebuilding stuff just once. > > I see that, I'm trying to avoid those package rebuilds even if it only > happens once. > > My question was: is there a benefit to not having "systemd" in > DISTRO_FEATURES all the time? for this I sent a patch [1] which just defines the proper distro features depending on the init system. I retested it but unfortunately test times remains basically the same so there is no much benefit; at least configuration string appending wont happen if patch is merged. [1] http://lists.openembedded.org/pipermail/openembedded-core/2017-April/136163.html