Bug 16440 - containers-library: Runqueue deadlocked on deferred tasks
Summary: containers-library: Runqueue deadlocked on deferred tasks
Status: NEW
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: oe-core other (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 6.2 M2
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2026-09-28 11:26 UTC by Mathieu Dubois-Briand
Modified: 2026-11-11 15:51 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Dubois-Briand 2026-09-28 11:26:18 UTC
As we now build containers regularly, we can see some warnings:

> WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_populate_sysroot
> WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_recipe_qa blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-core/gettext/gettext-minimal-native_1.0.bb:do_recipe_qa
> WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_populate_sysroot
> WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_recipe_qa blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.6.2.bb:do_recipe_qa
> WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:container-x86-64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/m4/m4-native_1.4.21.bb:do_populate_sysroot blocked by mc:container-aarch64:/srv/pokybuild/yocto-worker/containers-library/build/layers/openembedded-core/meta/recipes-devtools/m4/m4-native_1.4.21.bb:do_populate_sysroot
> ...
Comment 1 Mathieu Dubois-Briand 2026-09-28 11:53:11 UTC
containers-library debian12-vk-1 master completed at 2026-09-27 04:51:51+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/37/steps/20/logs/stdio

meta-virt debian12-vk-7 master&master-next completed at 2026-09-24 15:24:34+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4527/steps/15/logs/stdio

meta-virt opensuse160-vk-1 agodard/master-next completed at 2026-09-24 15:57:51+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4528/steps/15/logs/stdio

meta-virt ubuntu2404-vk-2 mathieu/master-next completed at 2026-09-27 07:44:31+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4537/steps/15/logs/stdio

meta-virt ubuntu2404-vk-1 mathieu/master-next completed at 2026-09-27 10:11:39+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4539/steps/15/logs/stdio

meta-virt debian11-vk-1 mathieu/master-next completed at 2026-09-27 16:59:57+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4540/steps/16/logs/stdio

meta-virt ubuntu2404-vk-3 mathieu/master-next completed at 2026-09-28 07:11:28+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4543/steps/15/logs/stdio
Comment 2 Mathieu Dubois-Briand 2026-09-30 09:05:59 UTC
containers-library debian12-vk-4 master completed at 2026-09-30 04:46:10+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/41/steps/20/logs/stdio

meta-virt stream9-vk-1 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-29 13:54:29+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4552/steps/16/logs/stdio

meta-virt ubuntu2510-vk-1 mathieu/master-next completed at 2026-09-29 14:21:54+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4553/steps/15/logs/stdio

meta-virt fedora44-vk-3 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-29 15:23:49+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4554/steps/15/logs/stdio

meta-virt debian12-vk-5 mathieu/master-next completed at 2026-09-30 06:34:02+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/89/builds/4559/steps/15/logs/stdio
Comment 3 Tim Orling 2026-10-01 15:18:15 UTC
These warnings have been there for a long time, but I never got anywhere trying to diagnose the problem. But as we are now building these on a regular basis, we should get to the bottom of it...
Comment 4 Bruce Ashfield 2026-10-01 18:04:16 UTC
Here's the details from my memory (which is a bit fuzzy now and
from some notes I found). I spent about 3 days trying to fix
these in meta-virt and failed. The warning / gate is harmless
in the end, and is a release valve, so I let it sit.

As we know m-virt reproduces this on multi-arch OCI container builds.
Setup:

- classes/oci-multiarch.bbclass (anon python ~line 94-103) wires
  mcdepends from the top-level multiarch recipe to each per-platform
  MC's do_image_complete:

    for platform in platforms.split():
        mc = d.getVarFlag('OCI_MULTIARCH_MC', platform)
        mcdepends.append(f"mc::{mc}:{recipe}:do_image_complete")
    d.setVarFlag('do_create_multiarch_index', 'mcdepends',
                 ' '.join(mcdepends))

- The two MCs pulled in are container-aarch64 and container-x86-64
  (conf/multiconfig/container-{aarch64,x86-64}.conf), both on the
  vcontainer distro.

- Reduced reproducer: `bitbake container-base-multiarch` with those
  two MCs active.

Observations from log analysis:

1. The deferred-and-then-forced TIDs always resolve to the same
   recipe, differing only in the mc: prefix: e.g.
   mc:container-x86-64:.../m4-native_1.4.21.bb:do_populate_sysroot
   vs mc:container-aarch64:.../m4-native_1.4.21.bb:do_populate_sysroot

2. Affected tasks cluster on bootstrap -native recipes
   (m4-native, libtool-native, gettext-minimal-native) and the two
   tasks are do_populate_sysroot and do_recipe_qa. Non-native
   recipes don't trigger it.

3. The forced-task output is correct across cross-checks. The two
   MCs' native sysroots are byte-identical to a single-MC build.
   That matches the theory that OEEquivHash correctly identifies the
   tasks as equivalent (both target the build host, same source,
   same -native tune), schedules both as deferred waiting for the
   other to produce the result, and the runqueue has to break the
   mutual-wait by forcing one.

4. The warning is persistent. clean builds, warm-sstate builds,
   and -c cleansstate+rebuild all produce it. Not transient.

Here's my memory of what I tried, they all broke a dependency
or some runnable output in my testing.

- "Shared-native builder MC" / multilib-flavored approach: dedicate a
  third multiconfig that owns the -native recipes and have the
  container-* MCs mcdepends-on it for their native sysroot. Fails
  because each target MC still parses -native recipes locally when a
  DEPENDS = "foo-native" exists in its graph, so the de-dup didn't
  actually happen. the warning persisted plus and I added machinery
  that wasn't earning its keep.

- MULTILIB_VARIANTS-style approach to pull -natives into a shared
  variant: multilib is designed around TARGET_* scopes, not
  BUILD_*; the sysroot discovery misaligned on native-vs-target.

- bbclass "trick" family: deltask the duplicate do_populate_sysroot
  on one MC and symlink the other MC's native sysroot in. Fragile
  on build-ordering (which MC's task "wins" varies by scheduler
  decisions you can't predict) and any recipe that direct-DEPENDS
  the native task fails parse.

- Pre-step warmup (bitbake <native-list> before the multi-MC build)
  was considered but rejected, not reachable through the recipe's
  own dependency chain, and goes stale on native-source bumps.

- Splitting the multi-arch build into separate top-level bitbake
  invocations (dropping mcdepends, stitching the index post-hoc)
  was considered but rejected on the error-prone / workflow-change
  grounds.

- Fixing it in bitbake's runqueue: teaching the defer/cover logic to
  recognize cross-MC -native equivalence as "either one runs, both are
  satisfied" rather than "one defers waiting for the other" .. is
  probably the right place for the fix, but is out of something
  I could hope to get right.

The current state: I document the warning as informational in my
cross-install troubleshooting notes:
"Runqueue deadlocked on deferred tasks" is bitbake's cross-MC dedup
safety valve firing, and doesn't affect the build outcome. The
forced output is byte-equal to a single-MC build.

That doesn't make it OK on the autobuilder side though. It's
noise on every multi-MC build and reads as a failure signal to
anyone unfamiliar with the pattern. If we can get it fixed at the
runqueue layer I'd prefer that; if there's an acceptable
hashequiv-level or scheduler-level nudge that avoids triggering the
defer-both path, happy to test it against our reproducer. I'd of
course also fix it on meta-virt, but I tried for some time, and
you have to be doing much more than just building a single
target to be sure it hasn't broken something else.

If helpful, I can produce a stripped-down meta-virt-reduced layer
exercising only oci-multiarch + two container-* MCs without the
rest of meta-virtualization, to make the reproducer small enough
to iterate on quickly.
Comment 5 Mathieu Dubois-Briand 2026-10-05 09:41:55 UTC
containers-library opensuse156-vk-1 master completed at 2026-10-04 05:01:15+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/42/steps/21/logs/stdio
Comment 6 Mathieu Dubois-Briand 2026-10-08 06:49:39 UTC
containers-library rocky10-vk-1 master completed at 2026-10-07 04:52:08+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/43/steps/20/logs/stdio
Comment 7 Mathieu Dubois-Briand 2026-10-21 07:47:25 UTC
containers-library debian12-vk-1 master completed at 2026-10-21 04:50:46+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/47/steps/20/logs/stdio
Comment 8 Mathieu Dubois-Briand 2026-10-29 08:54:07 UTC
containers-library debian12-vk-1 master completed at 2026-10-28 04:50:57+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/49/steps/20/logs/stdio
Comment 9 Mathieu Dubois-Briand 2026-11-02 07:45:56 UTC
containers-library debian12-vk-4 master completed at 2026-11-01 04:51:04+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/50/steps/20/logs/stdio
Comment 10 Mathieu Dubois-Briand 2026-11-04 08:40:48 UTC
containers-library alma8-vk-1 master completed at 2026-11-04 04:54:16+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/51/steps/21/logs/stdio
Comment 11 Mathieu Dubois-Briand 2026-11-09 08:53:58 UTC
containers-library alma10-vk-1 master completed at 2026-11-08 04:52:21+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/52/steps/20/logs/stdio
Comment 12 Mathieu Dubois-Briand 2026-11-11 15:51:26 UTC
containers-library debian13-vk-1 master completed at 2026-11-11 04:48:16+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/53/steps/20/logs/stdio