| Summary: | AB-INT: check-layer: unexplained changed signatures | ||
|---|---|---|---|
| Product: | [QA/Testing] Test Plans/Suite | Reporter: | Mathieu Dubois-Briand <mathieu.dubois-briand> |
| Component: | test-plans/suite | Assignee: | Paul Barker <paul> |
| Status: | NEW --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | paul, randy.macleod, richard.purdie, yoann.congal |
| Version: | unspecified | ||
| Target Milestone: | 6.2 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
check-layer alma8-vk-1 mathieu/master-next completed at 2026-05-11 08:32:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3797/steps/13/logs/stdio check-layer ubuntu2204-vk-1 master-next completed at 2026-05-13 14:11:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3811/steps/12/logs/stdio check-layer debian13-vk-2 master-next completed at 2026-05-14 00:57:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3814/steps/12/logs/stdio check-layer debian12-vk-3 mathieu/master-next completed at 2026-05-14 19:36:17+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3819/steps/12/logs/stdio check-layer ubuntu2404-vk-3 mathieu/master-next completed at 2026-05-15 16:55:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3839/steps/12/logs/stdio I added some debug code to master-next doing two things, firstly printing the exact signature differences and secondly preserving copies of all the locked sigs files. The build with the debug code failed here: https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3846 What is interesting is that there are just two signature differences listed: busybox:do_build changed 24e50a9a588e82050ef54ba288ac26e89b4b87e987685daacb06eb0a9dac1a3d -> c51657bb62a756c888d3acd1a5c6d9314cdc9aabc5e3c39ae1e717b73082bef5 busybox:do_populate_lic changed 4d27231de439cb257cf4ab552f5930f9fa9cd9ea33bbf9439af700c701f5cdc5 -> 363d7616636e03eb7bbfe2fae8da727cd16ca7610a8ee9bb2b937db36e6dbf65 do_build depends on do_populate_lic so that is expected. What is interesting is the do_populate_lic changes alone, no change in do_fetch. The diffsigs output is therefore misleading as not all those signatures actually changed, just do_populate_lic. Looking on disk at the failed build directory in ~/yocto-worker/check-layer/build/build/tmp/stamps/x86-64-v3-oe-linux/busybox/ we see: -rw-rw-r-- 1 pokybuild pokybuild 12608 May 19 00:19 1.38.0.do_populate_lic.sigdata.363d7616636e03eb7bbfe2fae8da727cd16ca7610a8ee9bb2b937db36e6dbf65 -rw-rw-r-- 1 pokybuild pokybuild 12558 May 18 23:07 1.38.0.do_populate_lic.sigdata.4d27231de439cb257cf4ab552f5930f9fa9cd9ea33bbf9439af700c701f5cdc5 decompressing these with zstd, the piping through jq to add line breaks, we can see: @@ -478,7 +478,7 @@ }, "taskhash": "4d27231de439cb257cf4ab552f5930f9fa9cd9ea33bbf9439af700c701f5cdc5", "taskhash_ignore_tasks": null, - "unihash": "363d7616636e03eb7bbfe2fae8da727cd16ca7610a8ee9bb2b937db36e6dbf65", + "unihash": "4d27231de439cb257cf4ab552f5930f9fa9cd9ea33bbf9439af700c701f5cdc5", "varvals": { "BP": "${BPN}-${PV}", "BPN": "${@oe.utils.prune_suffix(d.getVar('PN'), d.getVar('SPECIAL_PKGSUFFIX').split(), d)}", i.e. the taskhashes were the same but somehow a unihash match up was found. Given unihashes should be disabled in check-layer, this looks like the cause of the issues. The question therefore becomes, how is unihash somehow getting briefly enabled during the tests? check-layer opensuse160-vk-1 wrynose completed at 2026-05-19 16:38:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3851 check-layer ubuntu2404-vk-1 master-next&master completed at 2026-05-19 00:23:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3846/steps/12/logs/stdio check-layer opensuse156-vk-1 master completed at 2026-05-19 02:24:19+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3847/steps/13/logs/stdio check-layer debian12-vk-2 mathieu/master-next-tests&mathieu/master-next completed at 2026-06-15 09:24:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3997/steps/12/logs/stdio check-layer alma8-vk-2 master&master-next completed at 2026-06-12 14:55:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/3983/steps/13/logs/stdio check-layer debian12-vk-4 contrib/ycongal/wrynose-nut completed at 2026-06-18 17:21:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/27/builds/4025 check-layer alma8-vk-2 wrynose completed at 2026-07-04 14:25:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4138 check-layer ubuntu2204-vk-3 master&master-next completed at 2026-07-06 22:53:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4158/steps/12/logs/stdio check-layer debian12-vk-5 master&master-next completed at 2026-07-07 12:40:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4162/steps/12/logs/stdio check-layer alma8-vk-2 stable/2.18-nut&stable/wrynose-nut completed at 2026-07-04 14:25:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4138/steps/13/logs/stdio check-layer debian11-vk-1 mathieu/master-next completed at 2026-07-21 19:23:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4247/steps/13/logs/stdio check-layer alma10-vk-1 mathieu/master-next completed at 2026-08-19 10:23:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4494/steps/14/logs/stdio This was on a branch that failed to build, but looks unrelated to these failures. check-layer rocky10-vk-1 mathieu/master-next-tests&mathieu/master-next completed at 2026-08-26 19:56:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4563/steps/14/logs/stdio check-layer ubuntu2204-vk-1 wrynose completed at 2026-10-18 23:59:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4917 INFO: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/check-layer/build/layers/openembedded-core/scripts/lib/checklayer/cases/common.py", line 141, in test_signatures self.fail('Adding layer %s changed signatures.\n%s' % (self.tc.layer['name'], msg)) AssertionError: Adding layer meta changed signatures. 59 signatures changed, initial differences (first hash before, second after): busybox:do_recipe_qa: b7d75a012f50702647f54f4f2ca302a78368e9582af8160a91d07bd622a54e04 -> ae5900e39ac275822744f241eefc7f0e707e43e03fd3107d82b5190e4a4e6da9 bitbake-diffsigs --task busybox do_recipe_qa --signature b7d75a012f50702647f54f4f2ca302a78368e9582af8160a91d07bd622a54e04 ae5900e39ac275822744f241eefc7f0e707e43e03fd3107d82b5190e4a4e6da9 NOTE: Reconnecting to bitbake server... check-layer debian11-vk-3 mathieu/master-next completed at 2026-10-17 08:57:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4911/steps/15/logs/stdio check-layer stream9-vk-1 master completed at 2026-10-25 02:01:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4961/steps/15/logs/stdio A different but similar error now: check-layer rocky8-vk-1 mathieu/master-next-tests&mathieu/master-next completed at 2026-10-27 10:44:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4970/steps/15/logs/stdio check-layer fedora45-vk-1 master completed at 2026-10-28 01:42:58+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4974/steps/14/logs/stdio check-layer debian11-vk-1 blacksail completed at 2026-11-02 23:52:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/5005 | basehash changed from 63f194f6592c45ecc0fea61c19aaf35d00f6450266d773f00def99ac3a38f6c4 to 73cd7f03b0b23f4176562e05b09f3365d2e451187cec3143a0b0883726d93676 | Variable SRC_URI value changed: | "https://busybox.net/downloads/busybox-${PV}.tar.bz2;name=tarball [...] [-file://sha_accel.cfg-]" Note: sha_accel.cfg is added via a x86_64 override | meta/recipes-core/busybox/busybox_1.38.0.bb | 65:SRC_URI:append:x86-64 = " file://sha_accel.cfg" check-layer ubuntu2404-vk-3 master completed at 2026-11-08 01:44:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/5048/steps/18/logs/stdio check-layer rocky10-vk-1 master completed at 2026-11-11 02:09:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/5064/steps/14/logs/stdio |
We got a few failures of check-layer on both master-next and mathieu/master-next in the last days, and I can't understand what is the issue. INFO: test_signatures (common.CommonCheckLayer.test_signatures) INFO: ... FAIL Stdout: Loading cache...done. Loaded 1992 entries from dependency cache. INFO: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/check-layer/build/layers/openembedded-core/scripts/lib/checklayer/cases/common.py", line 141, in test_signatures self.fail('Adding layer %s changed signatures.\n%s' % (self.tc.layer['name'], msg)) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Adding layer meta changed signatures. 59 signatures changed, initial differences (first hash before, second after): busybox:do_recipe_qa: 7179671a25cf50a259e667e565b959c78711c629ff4882216a332426f3d6206e -> ae5900e39ac275822744f241eefc7f0e707e43e03fd3107d82b5190e4a4e6da9 bitbake-diffsigs --task busybox do_recipe_qa --signature 7179671a25cf50a259e667e565b959c78711c629ff4882216a332426f3d6206e ae5900e39ac275822744f241eefc7f0e707e43e03fd3107d82b5190e4a4e6da9 NOTE: Reconnecting to bitbake server... So first, I'm surprised as this is intermittent. Second, I believe I'm missing a point here: does changed signatures when "meta" is added make any sense? I understand "meta" as oe-core base layer, but without it we don't have any recipe, so how can we have signature changes?