We should improve the usability of yocto-check-layer by: * Using the existing bblayers.conf file to find dependency layer locations * Use bblayers.conf and existing layers to expand the layer under test ("meta-virtualization" -> full path) * Running the tests is an isolated separate build directory. This means layers don't need to be removed from bblayers.conf * By isolating to a separate build dir, having "virtualization" in DISTRO_FEATURES my local.conf wouldn't cause layer test failures with for example meta-virtualization
in other words, "yocto-check-layer meta-virtualization" in an existing build should be able to do the right things.
I'd also note that yocto-check-layer isn't working with BB_SERVER_TIMEOUT = 60: ====================================================================== INFO: FAIL: test_world_inherit_class (common.CommonCheckLayer.test_world_inherit_class) INFO: ---------------------------------------------------------------------- INFO: Traceback (most recent call last): File "/media/build/poky/build/layers/openembedded-core/scripts/lib/checklayer/cases/common.py", line 116, in test_world_inherit_class self.fail('\n'.join(msg)) ~~~~~~~~~^^^^^^^^^^^^^^^^ AssertionError: Layer meta-virtualization failed additional checks from yocto-check-layer.bbclass See below log for specific recipe parsing errors: Generating signatures failed. This might be due to some parse error and/or general layer incompatibilities. Command: BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS BB_SIGNATURE_HANDLER" BB_SIGNATURE_HANDLER="OEBasicHash" BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS INHERIT" INHERIT="yocto-check-layer" bitbake -S lockedsigs world Output: NOTE: Reconnecting to bitbake server... Loading cache...done. Loaded 5535 entries from dependency cache. NOTE: Resolving any missing task queue dependencies NOTE: Multiple providers are available for runtime ssh (dropbear, openssh) Consider defining a PREFERRED_RPROVIDER entry to match ssh NOTE: Multiple providers are available for runtime libnss-mdns (avahi-libnss-mdns, mdns) Consider defining a PREFERRED_RPROVIDER entry to match libnss-mdns Build Configuration: BB_VERSION = "2.19.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "x86_64-poky-linux" MACHINE = "qemux86-64" SDKMACHINE = "x86_64" DISTRO = "poky" DISTRO_VERSION = "5.3.99+snapshot-74e51ec95b08ba332b704f9053d25b0bf8a8ec0c" TUNE_FEATURES = "m64 x86-64-v3" meta meta-selftest = "master:74e51ec95b08ba332b704f9053d25b0bf8a8ec0c" meta-yocto-bsp meta-poky = "master:0998746f17032630563a0734360dd0f5bf07b1cb" meta-oe meta-filesystems meta-python meta-networking = "master:76700b6eaf327bb30e2c47b78713e2e409cfab7c" meta-virtualization = "master:2e0704ee9282e4e86c13180334cb7ce118e55dd9" Initialising tasks...Sstate summary: Wanted 33884 Local 17 Mirrors 0 Missed 33867 Current 21 (0% match, 0% complete) done. NOTE: Writing task signature files ERROR: Basehash mismatch cdc84650d6c3ba6f438e20ffd9592980ca4f7d660cc5020e665005f8cc9a2e94 versus fbd45000b8513e82e9febcaa993ae670a25cbb95292b5e99f6e35416a4725d11 for /media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_recipe_qa ERROR: Basehash mismatch 1b0a17ed68d372155434dc08cd47a2ff7cd5e3b2c29e42df7d6f138c3086ebc4 versus 12783a7b67fa9ae8f5b8064efdc12af763d9b25bdace4fb35c593b5f4ce73d81 for /media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_populate_lic ERROR: Basehash mismatch b6fb4a5044c5a43b37375cd6b07bbda02e4314a1da77632018d8132eb2ccdd67 versus 533af2e40b7d7e9cc05e657b5f3e1e34c5b133a05df518020b484efe6a148158 for /media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_create_spdx ERROR: Basehash mismatch e47aa5fd2004b73168f84b8a3de03cfa55f9d2f40e8e2ebba140d41948f97297 versus 6c26963fa9ce7e343b79b3a825285645442bfcc2dc95561dae2cfca101b47de6 for /media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_fetch ERROR: Basehash mismatch d35ef79a15950ae50ae3becacf4ccd459ae9e43984d872189fcafebe6215da98 versus e3961102e2695a304968c168728896f1266805bf4198c3af5de826964c2edcb1 for /media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_unpack ERROR: Basehash mismatch 1d63362a2c94ab99b0c54efad21de54c1d35b81c980ce1817b0a2131573715e7 versus acb6a0e7e9e564357354732a4f8952936cbde4f83e36ab3c73b8969c0228d639 for /media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_install ERROR: Basehash mismatch 591fd1e523f55fd5a363f8b1203b9e1b529588e6404c625b2b4ed95eb5ca08e3 versus 306d0466d1cadedd01adc3702ecbafdad372dc047720a7edb15281343e28b0fd for virtual:native:/media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_fetch ERROR: Basehash mismatch 104fc3c807fd091cbd46881cc3a887090e006d46308eed21c40886158539057f versus f056ec84db63adc042117022be9135dfbb9c729687ee8d63be1249550643154a for virtual:native:/media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_unpack ERROR: Basehash mismatch e508899183bc5f7bfb6eeb30297671db86df964d6b771345709bc8ba92ca5f5f versus 90a760d68d7c9e6ba523f5645054c1f221cade6549a3d38c3a44986a5a01d7d4 for virtual:native:/media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_install ERROR: Basehash mismatch 06257696142c0fd429e609457742a59ca55e5d887398b598091d5bcdc2e8aa39 versus 3bed63b42b0ac209663fd795bd639e17be23cddb58efac60ca5365fade282de4 for virtual:native:/media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_create_spdx ERROR: Basehash mismatch 980f204631cc3c94ccfac815b942828bfb0dd63ed988ea902e3faecb326f686d versus 1309bd1a1f0f6babba8689e5e35358119873ab2f72fd44a16c361aa6664f4c5d for virtual:native:/media/build/poky/build/layers/openembedded-core/meta/recipes-support/ca-certificates/ca-certificates_20260223.bb:do_recipe_qa Writing locked sigs to /media/build/poky/build/build/locked-sigs.inc NOTE: Tasks Summary: Attempted 0 tasks of which 0 didn't need to be rerun and all succeeded.
Additionally, * We should enumerate the machines in a BSP layer instead of needing them to be provided on the command line. * We should identify and handle "bare-metal" machines which are not expected to be the target for a full Linux system build, e.g. meta-ti has some of these which currently cause yocto-check-layer failures if you try to include them in the list of machines to test. * The test_machine_signatures case should be extended to support non-BSP layers, using at least two machines from openembedded-core to catch things like machine-specific recipes where PACKAGE_ARCH is not set correctly, allarch recipes which are really tune/machine specific, etc.
Oh, and multi-config support (may already be written down elsewhere but putting it here so we don't forget!)
There's also https://bugzilla.yoctoproject.org/show_bug.cgi?id=14148 which points out that the argument parser is full of ambiguous options and making it do the right thing is non-obvious.