Bug 16170 - yocto-check-layer: test never ends on broken layers
Summary: yocto-check-layer: test never ends on broken layers
Status: RESOLVED FIXED
Alias: None
Product: Tooling
Classification: QA/Testing
Component: tooling (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High normal
Target Milestone: 6.0 M3
Assignee: Tim Orling
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2026-02-12 12:18 UTC by Mathieu Dubois-Briand
Modified: 2026-02-19 15:42 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Dubois-Briand 2026-02-12 12:18:08 UTC
In some conditions, yocto-check-layer common.CommonCheckLayer.test_signatures test never finishes or at least not in a reasonable amount of time.

This is seen on the autobuilder with meta-virtualization:
> INFO: test_signatures (common.CommonCheckLayer)
> 
> command timed out: 16200 seconds without output running [b'/srv/pokybuild/yocto-worker/check-layer-nightly/yocto-autobuilder-helper/scripts/run-config', b'check-layer-nightly', b'/srv/pokybuild/yocto-worker/check-layer-nightly/build/build', b'master', b'ssh://git@push.yoctoproject.org/poky', b'--sstateprefix', b'', b'--buildappsrcrev', b'', b'--publish-dir', b'', b'--build-type', b'quick', b'--workername', b'debian11-vk-1', b'--build-url', b'https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/976', b'--results-dir', b'/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20260211-3/testresults', b'--quietlogging', b'--stepname', b'cmds', b'--phase', b'7'], attempting to kill
> process killed by signal 9

But it can also easily be reproduced locally with:
> yocto-check-layer-wrapper ../meta-virtualization --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python ../meta-openembedded/meta-filesystems --no-auto-dependency

We know that current master branch on meta-virtualization is broken, but yocto-check-layer should still report errors instead of being stuck.

Breaking commit in meta-virtualization: https://git.yoctoproject.org/meta-virtualization/commit/?id=6f53d7763906f24d86e89da4da744421c52d591c

Code is stuck get_depgraph() while waiting for tifoil to finish running the "generateDepTreeEvent" command. I added some traces: we are receiving events, but none allowing to get out of the "while" loop.
https://git.openembedded.org/openembedded-core/tree/scripts/lib/checklayer/__init__.py#n367

Looking at bitbake-cookerdaemon.log, it looks like the command did finish and some reply was sent. So probably some logic is wrong somewhere in interpreting this rely.
> 2149754 13:14:50.133747 Running command ['generateDepTreeEvent', ['world'], 'do_build']
> 2149754 13:14:50.133787 Registering idle function <bound method Command.runAsyncCommand of <bb.command.Command object at 0x7f66dfe61fd0>>
> 2149754 13:14:50.133794 Sending reply (True, None)
> 2149754 13:14:50.133827 Command Completed (socket: True)
> 2149754 13:14:51.133844 Parsing completed
> 2149754 13:14:52.380348 Removing idle function <bound method Command.runAsyncCommand of <bb.command.Command object at 0x7f66dfe61fd0>> at idleFinish

Stopping investigations here for now.
Comment 1 Mathieu Dubois-Briand 2026-02-12 12:19:18 UTC
check-layer-nightly debian11-vk-1 master completed at 2026-02-11 06:49:50+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/976/steps/24/logs/stdio

check-layer-nightly debian13-vk-1 master completed at 2026-02-12 05:29:53+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/977/steps/24/logs/stdio
Comment 2 Randy MacLeod 2026-02-12 15:51:33 UTC
Two issues:
 1. meta-virt should pass (M+)
 2. yocto-check-layer should not hang (High).
Comment 3 Bruce Ashfield 2026-02-12 16:13:14 UTC
FWIW. I'm seeing the same thing on my local runs, it hangs at the same point and never continues.

Whether I use the wrapper, the tool directly, etc, has no impact. But I can run the bitbake commands directly and they complete in the expected amount of time.

I found there were some pipe and other issues and maybe a timetout involved, but ran out of time to debug it.
Comment 6 Mathieu Dubois-Briand 2026-02-16 11:47:27 UTC
check-layer-nightly debian12-vk-7 master completed at 2026-02-13 05:30:23+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/980/steps/24/logs/stdio

check-layer-nightly ubuntu2204-vk-4 master completed at 2026-02-14 05:40:47+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/982/steps/24/logs/stdio

check-layer-nightly debian12-vk-7 master completed at 2026-02-15 05:30:19+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/983/steps/24/logs/stdio

check-layer-nightly ubuntu2404-vk-2 master completed at 2026-02-16 05:33:27+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/987/steps/24/logs/stdio
Comment 7 Tim Orling 2026-02-17 23:01:49 UTC
These most recent runs are without my patch applied.
Comment 8 Mathieu Dubois-Briand 2026-02-19 08:38:34 UTC
The patch is still not applied on master, so following failures are expected:

check-layer-nightly fedora40-vk-4 master completed at 2026-02-19 05:38:38+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/993/steps/24/logs/stdio

check-layer-nightly debian12-vk-5 master completed at 2026-02-17 05:30:27+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/990/steps/24/logs/stdio

check-layer-nightly rocky9-vk-1 master completed at 2026-02-18 08:23:18+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/85/builds/992/steps/24/logs/stdio