| Summary: | util-linux ptest results are inconsistent per image | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Joshua Watt <JPEWhacker> | ||||||
| Component: | core | Assignee: | Mariano Lopez <just.another.mariano> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | Medium+ | CC: | alejandr, bruce.ashfield, just.another.mariano, meta.mr.watcher, meta.watcher, randy.macleod, richard.purdie | ||||||
| Version: | 2.7 | ||||||||
| Target Milestone: | 2.8 M1 | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | |||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||
| Attachments: |
|
||||||||
|
Description
Joshua Watt
2019-04-18 18:40:40 UTC
Created attachment 4505 [details]
ptest log from core-image-sato build
Created attachment 4506 [details]
ptest log from core-image-sato-sdk build
When I was working on ptest for util-linux I found the package's tests is divided by tests and sub-tests and this is by design by upstream. Some of these tests will check for a binary in order to run, if the binary is not in the image then the test along with the sub-tests won't run. The reason the numbers don't add up is that when a test is skipped it won't report all the sub-tests that were skipped, just the "main" test. For example, check the low-probe test which is on the "usr/lib/util-linux/ptest/tests/ts/blkid/low-probe" path and reported as "blkid: superblocks probing". In case the test is executed (it requires xz), it will run 78 sub-tests, all of these tests will report a "PASS" that will add up in the "Passed" tests, in case the test is not executed a single "SKIP" will be reported. I think this is not a bug, just the lack of consistency on how the skipped tests are reported. That makes sense and explains why the skipped counts are incorrect. Shouldn't the dependencies of the util-linux-ptest package pull in the binaries we need to test though which would make the pass counts consistent across images? I was checking all dependencies for the ptest of util-linux and I need some guidance on what should be enabled and what we should keep disabled for a couple of packages: 1. mdadm: As far as I know, currently the mdadm's tests are failing. The specific use for the util-linux's test is to create/delete disks. 2. scsi_debug kernel module: Used also to create/delete disks. In case this should be added to the dependencies, should I add the configuration option in the yocto-kernel-cache repo? I have identified the dependencies needed to have the same amount of tests to be run regardless the image type. I can submit that patch now and continue working with the rest if that would help to have better util-linux ptest baseline for the current/next releases. Please let me know your comments. Checking why the libmount tests are skipped I found the tests aren't built in the first place because we are building shared libraries instead of static ones. You can check this on line 1076 of the configure.ac file: AM_CONDITIONAL([BUILD_LIBMOUNT_TESTS], [test "x$build_libmount" = xyes -a "x$enable_static" = xyes]) Reset from 0.0.0 to 2.8 M1. The patch to add the missing dependencies of util-linux-ptest package has been integrated into master: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e42d45925d4a6be1d7cb240c0b630c7529d5a7cc The only missing thing is the kernel module listed in RRECOMMENDS. Currently, the module is not built (therefore not installed), and the yocto-kernel-cache repository doesn't have an option to enable the module. I think is a good idea having the configuration options in the yocto-kernel-cache repository because this would allow continuing with the automated testing. Should we group such configuration in a "ptest bundle"? If so, in what part of the tree should be in? At this point, it seems the only ptest package that requires a kernel module not installed by default is util-linux. Perhaps in the future other test packages would require another kernel module. This is why I'm thinking of a bundle of modules that should be built when ptest is enabled. A couple of years ago, we did a split between a production and debug/test kernel type. So it isn't out of the question that we could enable the missing options in the debug kernel by default. As long as they aren't too many, or don't bloat the kernel size, it shouldn't be a problem. Alternatively, there could be some distro feature that linux yocto picks up on, and includes the extra fragments to enable the missing options. The distro feature seems more precise/clean to me, but I'm not sure if that is acceptable to everyone. I think the DISTRO_FEATURE would be the way to go, it is cleaner and it would be a step forward into allowing us to do this for more packages if required. We could check if ptest is enabled? (In reply to comment #11) > We could check if ptest is enabled? Exactly, I was thinking on something like: ${@bb.utils.contains("DISTRO_FEATURES", "ptest", " features/scsi/scsi-debug.scc", "" ,d)}" I sent the yocto-kernel-cache patch and waiting for review: https://lists.yoctoproject.org/pipermail/linux-yocto/2019-June/007738.html Thanks, Alex, Bruce, and Richard! The cache piece merged... Last part is merged into master: https://git.openembedded.org/openembedded-core/commit/?id=70a30872bd93cc058b05d9cf2b4f9334658629ee Thanks, Alex, Bruce, and Richard for your support on this one! |