16440 2026-09-28 11:26:18 +0000 containers-library: Runqueue deadlocked on deferred tasks 2026-11-11 15:51:26 +0000 1 1 7 Build System, Metadata & Runtime OE-Core oe-core other unspecified x86 Multiple NEW Medium+ normal 6.2 M2 1 mathieu.dubois-briand richard.purdie bruce.ashfield randy.macleod tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 106947 0 mathieu.dubois-briand 2026-09-28 11:26:18 +0000 As we now build containers regularly, we can see some warnings: > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_populate_sysroot > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_recipe_qa blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_recipe_qa > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_populate_sysroot > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_recipe_qa blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_recipe_qa > WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/m4/m4-native_1.4.21.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/m4/m4-native_1.4.21.bb:do_populate_sysroot > ... 106950 1 mathieu.dubois-briand 2026-09-28 11:53:11 +0000 containers-library debian12-vk-1 master completed at 2026-09-27 04:51:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/37/steps/20/logs/stdio meta-virt debian12-vk-7 master&master-next completed at 2026-09-24 15:24:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4527/steps/15/logs/stdio meta-virt opensuse160-vk-1 agodard/master-next completed at 2026-09-24 15:57:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4528/steps/15/logs/stdio meta-virt ubuntu2404-vk-2 mathieu/master-next completed at 2026-09-27 07:44:31+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4537/steps/15/logs/stdio meta-virt ubuntu2404-vk-1 mathieu/master-next completed at 2026-09-27 10:11:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4539/steps/15/logs/stdio meta-virt debian11-vk-1 mathieu/master-next completed at 2026-09-27 16:59:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4540/steps/16/logs/stdio meta-virt ubuntu2404-vk-3 mathieu/master-next completed at 2026-09-28 07:11:28+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4543/steps/15/logs/stdio 106965 2 mathieu.dubois-briand 2026-09-30 09:05:59 +0000 containers-library debian12-vk-4 master completed at 2026-09-30 04:46:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/41/steps/20/logs/stdio meta-virt stream9-vk-1 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-29 13:54:29+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4552/steps/16/logs/stdio meta-virt ubuntu2510-vk-1 mathieu/master-next completed at 2026-09-29 14:21:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4553/steps/15/logs/stdio meta-virt fedora44-vk-3 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-29 15:23:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4554/steps/15/logs/stdio meta-virt debian12-vk-5 mathieu/master-next completed at 2026-09-30 06:34:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4559/steps/15/logs/stdio 106985 3 tim.orling 2026-10-01 15:18:15 +0000 These warnings have been there for a long time, but I never got anywhere trying to diagnose the problem. But as we are now building these on a regular basis, we should get to the bottom of it... 106993 4 bruce.ashfield 2026-10-01 18:04:16 +0000 Here's the details from my memory (which is a bit fuzzy now and from some notes I found). I spent about 3 days trying to fix these in meta-virt and failed. The warning / gate is harmless in the end, and is a release valve, so I let it sit. As we know m-virt reproduces this on multi-arch OCI container builds. Setup: - classes/oci-multiarch.bbclass (anon python ~line 94-103) wires mcdepends from the top-level multiarch recipe to each per-platform MC's do_image_complete: for platform in platforms.split(): mc = d.getVarFlag('OCI_MULTIARCH_MC', platform) mcdepends.append(f"mc::{mc}:{recipe}:do_image_complete") d.setVarFlag('do_create_multiarch_index', 'mcdepends', ' '.join(mcdepends)) - The two MCs pulled in are container-aarch64 and container-x86-64 (conf/multiconfig/container-{aarch64,x86-64}.conf), both on the vcontainer distro. - Reduced reproducer: `bitbake container-base-multiarch` with those two MCs active. Observations from log analysis: 1. The deferred-and-then-forced TIDs always resolve to the same recipe, differing only in the mc: prefix: e.g. mc:container-x86-64:.../m4-native_1.4.21.bb:do_populate_sysroot vs mc:container-aarch64:.../m4-native_1.4.21.bb:do_populate_sysroot 2. Affected tasks cluster on bootstrap -native recipes (m4-native, libtool-native, gettext-minimal-native) and the two tasks are do_populate_sysroot and do_recipe_qa. Non-native recipes don't trigger it. 3. The forced-task output is correct across cross-checks. The two MCs' native sysroots are byte-identical to a single-MC build. That matches the theory that OEEquivHash correctly identifies the tasks as equivalent (both target the build host, same source, same -native tune), schedules both as deferred waiting for the other to produce the result, and the runqueue has to break the mutual-wait by forcing one. 4. The warning is persistent. clean builds, warm-sstate builds, and -c cleansstate+rebuild all produce it. Not transient. Here's my memory of what I tried, they all broke a dependency or some runnable output in my testing. - "Shared-native builder MC" / multilib-flavored approach: dedicate a third multiconfig that owns the -native recipes and have the container-* MCs mcdepends-on it for their native sysroot. Fails because each target MC still parses -native recipes locally when a DEPENDS = "foo-native" exists in its graph, so the de-dup didn't actually happen. the warning persisted plus and I added machinery that wasn't earning its keep. - MULTILIB_VARIANTS-style approach to pull -natives into a shared variant: multilib is designed around TARGET_* scopes, not BUILD_*; the sysroot discovery misaligned on native-vs-target. - bbclass "trick" family: deltask the duplicate do_populate_sysroot on one MC and symlink the other MC's native sysroot in. Fragile on build-ordering (which MC's task "wins" varies by scheduler decisions you can't predict) and any recipe that direct-DEPENDS the native task fails parse. - Pre-step warmup (bitbake <native-list> before the multi-MC build) was considered but rejected, not reachable through the recipe's own dependency chain, and goes stale on native-source bumps. - Splitting the multi-arch build into separate top-level bitbake invocations (dropping mcdepends, stitching the index post-hoc) was considered but rejected on the error-prone / workflow-change grounds. - Fixing it in bitbake's runqueue: teaching the defer/cover logic to recognize cross-MC -native equivalence as "either one runs, both are satisfied" rather than "one defers waiting for the other" .. is probably the right place for the fix, but is out of something I could hope to get right. The current state: I document the warning as informational in my cross-install troubleshooting notes: "Runqueue deadlocked on deferred tasks" is bitbake's cross-MC dedup safety valve firing, and doesn't affect the build outcome. The forced output is byte-equal to a single-MC build. That doesn't make it OK on the autobuilder side though. It's noise on every multi-MC build and reads as a failure signal to anyone unfamiliar with the pattern. If we can get it fixed at the runqueue layer I'd prefer that; if there's an acceptable hashequiv-level or scheduler-level nudge that avoids triggering the defer-both path, happy to test it against our reproducer. I'd of course also fix it on meta-virt, but I tried for some time, and you have to be doing much more than just building a single target to be sure it hasn't broken something else. If helpful, I can produce a stripped-down meta-virt-reduced layer exercising only oci-multiarch + two container-* MCs without the rest of meta-virtualization, to make the reproducer small enough to iterate on quickly. 107010 5 mathieu.dubois-briand 2026-10-05 09:41:55 +0000 containers-library opensuse156-vk-1 master completed at 2026-10-04 05:01:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/42/steps/21/logs/stdio 107028 6 mathieu.dubois-briand 2026-10-08 06:49:39 +0000 containers-library rocky10-vk-1 master completed at 2026-10-07 04:52:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/43/steps/20/logs/stdio 107147 7 mathieu.dubois-briand 2026-10-21 07:47:25 +0000 containers-library debian12-vk-1 master completed at 2026-10-21 04:50:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/47/steps/20/logs/stdio 107227 8 mathieu.dubois-briand 2026-10-29 08:54:07 +0000 containers-library debian12-vk-1 master completed at 2026-10-28 04:50:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/49/steps/20/logs/stdio 107266 9 mathieu.dubois-briand 2026-11-02 07:45:56 +0000 containers-library debian12-vk-4 master completed at 2026-11-01 04:51:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/50/steps/20/logs/stdio 107293 10 mathieu.dubois-briand 2026-11-04 08:40:48 +0000 containers-library alma8-vk-1 master completed at 2026-11-04 04:54:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/51/steps/21/logs/stdio 107345 11 mathieu.dubois-briand 2026-11-09 08:53:58 +0000 containers-library alma10-vk-1 master completed at 2026-11-08 04:52:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/52/steps/20/logs/stdio 107377 12 mathieu.dubois-briand 2026-11-11 15:51:26 +0000 containers-library debian13-vk-1 master completed at 2026-11-11 04:48:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/53/steps/20/logs/stdio