For testing complex boot scenarios (like TPM, image-installer, etc.) we need control over booting qemu. Such testing is easy in a test based on oe-selftest and impossible with testimage.bbclass based tests, so we need to extend the refkit CI so that it invokes oe-selftest with a certain list of tests. It might be useful to have two sets: * One which is used before building. If one of these tests fails, building can be skipped because there is a fatal flaw. * One which is used after building, but only if building succeeded. These tests then can reuse build artifacts (like images) if they exist already. The sstate test that we have falls into the first set, the (to be written) image tests into the second. The sets could be defined in the CI .inc file.
refkit CI currently does not run iotsstatetests.SStateTests.test_sstate_samesigs before building. Could it add any value running that test in refkit case?
(In reply to comment #1) > refkit CI currently does not run > iotsstatetests.SStateTests.test_sstate_samesigs > before building. > Could it add any value running that test in refkit case? The test would need to be updated to reflect the current set of supported machines, but yes, in principle it might still be useful.
tried in refkit current master, takes 1min on a desktop machine. Indeed, MACHINE set needs editing in meta-refkit/lib/oeqa/selftest/iotsstatetests.py Is the idea that we keep changing that file, or should we attempt defining machine set outside of code, in some CI_ variables?
or another idea, can we modify samesigs test so that it takes MACHINES as argument? In refkit CI, machine set to build is defined in Jenkinsfile so most convenient would be just run samesigs giving MACHINES as argument, instead of parsing machines from environment or strings defined in files etc.
(In reply to comment #4) > or another idea, can we modify samesigs test so that it takes > MACHINES as argument? Yes, that would be useful. I also need to check whether improvements that I made to that test could go back to OE-core. I did something (better error reporting?) but without going back to the Ostro repo I am not sure what those were. But enabling the mechanism for running oe-selftest is a bit orthogonal to actually enabling tests which use the mechanism. I'm saying "a bit" because obviously some tests are needed for testing the mechanism ;-}
Stephen, I think we need to keep this assigned to Olev. We need this enabled in the current Refkit CI scripts for Jenkins, and I believe Olev is still the one maintaining those.
Let's keep this ticket for enabling running additional tests in CI, (both pre-build and after build), and create a separate ticket about creating after-build, runnable-in-qemu, tests.
The goal has always been to focus on "enable oe-selftest tests" (not writing them) and "extend the refkit CI so that it invokes oe-selftest with a certain list of tests". So yes, this issue here is about the CI system, not the tests using that functionality. Those were just mentioned to give a justification why such a capability in the CI system are needed, and why two slots are useful. Also note that running tests works best when they use exactly the same build configuration as the build itself (better sstate reuse). This influences how to hook the tests into the CI system.
Refkit PR#53 has draft implementation of both pre-build and after-build selftest activation. You can check the last of 6 commits only for that, others are about Jenkinsfile refactoring, as I started to prepare for separate stage implementation for running test. Later, plan changed and test calls were inserted into build script. Test names are taken from REFKIT_CI_ variables. As pre-build, current sstate-samesigs test is used. As post-build, there are no tests yet, I verified with samesigs test that step runs.
PR#53 now reached a next checkpoint state, please check. Per Mikko's idea, I moved BUILDHISTORY enablement from refkit-ci.inc to build-project.sh which allows to simplify conf/auto.conf creation during oe-elftest. pre- and post- tests are now as separate scripts and they will show as separate CI build phases, to allow clear status reporting from stages. Overall, this change also caused examination of other things which leads to multiple commits with cleanup and re-structure. for example, we were still running several CI-only-related steps in local build, which is avoided in PR#53 current state of code.
PR#53 was merged, CI can now run oe-selftests, both pre- and post-build