| Summary: | containers-library: Runqueue deadlocked on deferred tasks | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Mathieu Dubois-Briand <mathieu.dubois-briand> |
| Component: | oe-core other | Assignee: | Richard Purdie <richard.purdie> |
| Status: | NEW --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | bruce.ashfield, randy.macleod, tim.orling |
| Version: | unspecified | ||
| Target Milestone: | 6.2 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
containers-library debian12-vk-1 master completed at 2026-09-27 04:51:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/37/steps/20/logs/stdio meta-virt debian12-vk-7 master&master-next completed at 2026-09-24 15:24:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4527/steps/15/logs/stdio meta-virt opensuse160-vk-1 agodard/master-next completed at 2026-09-24 15:57:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4528/steps/15/logs/stdio meta-virt ubuntu2404-vk-2 mathieu/master-next completed at 2026-09-27 07:44:31+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4537/steps/15/logs/stdio meta-virt ubuntu2404-vk-1 mathieu/master-next completed at 2026-09-27 10:11:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4539/steps/15/logs/stdio meta-virt debian11-vk-1 mathieu/master-next completed at 2026-09-27 16:59:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4540/steps/16/logs/stdio meta-virt ubuntu2404-vk-3 mathieu/master-next completed at 2026-09-28 07:11:28+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4543/steps/15/logs/stdio containers-library debian12-vk-4 master completed at 2026-09-30 04:46:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/41/steps/20/logs/stdio meta-virt stream9-vk-1 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-29 13:54:29+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4552/steps/16/logs/stdio meta-virt ubuntu2510-vk-1 mathieu/master-next completed at 2026-09-29 14:21:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4553/steps/15/logs/stdio meta-virt fedora44-vk-3 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-29 15:23:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4554/steps/15/logs/stdio meta-virt debian12-vk-5 mathieu/master-next completed at 2026-09-30 06:34:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4559/steps/15/logs/stdio These warnings have been there for a long time, but I never got anywhere trying to diagnose the problem. But as we are now building these on a regular basis, we should get to the bottom of it... Here's the details from my memory (which is a bit fuzzy now and
from some notes I found). I spent about 3 days trying to fix
these in meta-virt and failed. The warning / gate is harmless
in the end, and is a release valve, so I let it sit.
As we know m-virt reproduces this on multi-arch OCI container builds.
Setup:
- classes/oci-multiarch.bbclass (anon python ~line 94-103) wires
mcdepends from the top-level multiarch recipe to each per-platform
MC's do_image_complete:
for platform in platforms.split():
mc = d.getVarFlag('OCI_MULTIARCH_MC', platform)
mcdepends.append(f"mc::{mc}:{recipe}:do_image_complete")
d.setVarFlag('do_create_multiarch_index', 'mcdepends',
' '.join(mcdepends))
- The two MCs pulled in are container-aarch64 and container-x86-64
(conf/multiconfig/container-{aarch64,x86-64}.conf), both on the
vcontainer distro.
- Reduced reproducer: `bitbake container-base-multiarch` with those
two MCs active.
Observations from log analysis:
1. The deferred-and-then-forced TIDs always resolve to the same
recipe, differing only in the mc: prefix: e.g.
mc:container-x86-64:.../m4-native_1.4.21.bb:do_populate_sysroot
vs mc:container-aarch64:.../m4-native_1.4.21.bb:do_populate_sysroot
2. Affected tasks cluster on bootstrap -native recipes
(m4-native, libtool-native, gettext-minimal-native) and the two
tasks are do_populate_sysroot and do_recipe_qa. Non-native
recipes don't trigger it.
3. The forced-task output is correct across cross-checks. The two
MCs' native sysroots are byte-identical to a single-MC build.
That matches the theory that OEEquivHash correctly identifies the
tasks as equivalent (both target the build host, same source,
same -native tune), schedules both as deferred waiting for the
other to produce the result, and the runqueue has to break the
mutual-wait by forcing one.
4. The warning is persistent. clean builds, warm-sstate builds,
and -c cleansstate+rebuild all produce it. Not transient.
Here's my memory of what I tried, they all broke a dependency
or some runnable output in my testing.
- "Shared-native builder MC" / multilib-flavored approach: dedicate a
third multiconfig that owns the -native recipes and have the
container-* MCs mcdepends-on it for their native sysroot. Fails
because each target MC still parses -native recipes locally when a
DEPENDS = "foo-native" exists in its graph, so the de-dup didn't
actually happen. the warning persisted plus and I added machinery
that wasn't earning its keep.
- MULTILIB_VARIANTS-style approach to pull -natives into a shared
variant: multilib is designed around TARGET_* scopes, not
BUILD_*; the sysroot discovery misaligned on native-vs-target.
- bbclass "trick" family: deltask the duplicate do_populate_sysroot
on one MC and symlink the other MC's native sysroot in. Fragile
on build-ordering (which MC's task "wins" varies by scheduler
decisions you can't predict) and any recipe that direct-DEPENDS
the native task fails parse.
- Pre-step warmup (bitbake <native-list> before the multi-MC build)
was considered but rejected, not reachable through the recipe's
own dependency chain, and goes stale on native-source bumps.
- Splitting the multi-arch build into separate top-level bitbake
invocations (dropping mcdepends, stitching the index post-hoc)
was considered but rejected on the error-prone / workflow-change
grounds.
- Fixing it in bitbake's runqueue: teaching the defer/cover logic to
recognize cross-MC -native equivalence as "either one runs, both are
satisfied" rather than "one defers waiting for the other" .. is
probably the right place for the fix, but is out of something
I could hope to get right.
The current state: I document the warning as informational in my
cross-install troubleshooting notes:
"Runqueue deadlocked on deferred tasks" is bitbake's cross-MC dedup
safety valve firing, and doesn't affect the build outcome. The
forced output is byte-equal to a single-MC build.
That doesn't make it OK on the autobuilder side though. It's
noise on every multi-MC build and reads as a failure signal to
anyone unfamiliar with the pattern. If we can get it fixed at the
runqueue layer I'd prefer that; if there's an acceptable
hashequiv-level or scheduler-level nudge that avoids triggering the
defer-both path, happy to test it against our reproducer. I'd of
course also fix it on meta-virt, but I tried for some time, and
you have to be doing much more than just building a single
target to be sure it hasn't broken something else.
If helpful, I can produce a stripped-down meta-virt-reduced layer
exercising only oci-multiarch + two container-* MCs without the
rest of meta-virtualization, to make the reproducer small enough
to iterate on quickly.
containers-library opensuse156-vk-1 master completed at 2026-10-04 05:01:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/42/steps/21/logs/stdio containers-library rocky10-vk-1 master completed at 2026-10-07 04:52:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/43/steps/20/logs/stdio containers-library debian12-vk-1 master completed at 2026-10-21 04:50:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/47/steps/20/logs/stdio containers-library debian12-vk-1 master completed at 2026-10-28 04:50:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/49/steps/20/logs/stdio containers-library debian12-vk-4 master completed at 2026-11-01 04:51:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/50/steps/20/logs/stdio containers-library alma8-vk-1 master completed at 2026-11-04 04:54:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/51/steps/21/logs/stdio containers-library alma10-vk-1 master completed at 2026-11-08 04:52:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/52/steps/20/logs/stdio containers-library debian13-vk-1 master completed at 2026-11-11 04:48:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/53/steps/20/logs/stdio |
As we now build containers regularly, we can see some warnings: > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_populate_sysroot > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_recipe_qa blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_recipe_qa > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_populate_sysroot > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_recipe_qa blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_recipe_qa > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/m4/m4-native_1.4.21.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/m4/m4-native_1.4.21.bb:do_populate_sysroot > ...