Bug 16080

Summary: bitbake: fork() use in multi-threaded warning
Product: [Build System, Metadata & Runtime] BitBake Reporter: Mathieu Dubois-Briand <mathieu.dubois-briand>
Component: bitbakeAssignee: Mathieu Dubois-Briand <mathieu.dubois-briand>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: ccasciato, poky.bs.watcher, poky.watcher, Qi.Chen, randy.macleod
Version: unspecified   
Target Milestone: 6.0   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Mathieu Dubois-Briand 2025-11-27 08:59:09 UTC
We got this warning on a few workers:

WARNING: /srv/pokybuild/yocto-worker/qemuarm64-armhost/build/layers/bitbake/lib/bb/daemonize.py:37: DeprecationWarning: This process (pid=1039305) is multi-threaded, use of fork() may lead to deadlocks in the child.
Comment 1 Mathieu Dubois-Briand 2025-11-27 09:01:00 UTC
qemuarm64-armhost debian13-vk-arm1 master-next completed at 2025-11-20 08:34:09+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/2774/steps/12/logs/stdio

qemux86-alt opensuse155-vk-2 master-next completed at 2025-11-21 11:03:49+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/2733/steps/15/logs/stdio

qemuarm64-armhost debian13-vk-arm1 master-next completed at 2025-11-21 13:52:29+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/2783/steps/12/logs/stdio

qemuarm-alt rocky8-vk-1 master completed at 2025-11-26 09:16:28+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/74/builds/2761/steps/14/logs/stdio

qemuarm64-armhost debian13-vk-arm1 master completed at 2025-11-27 01:37:05+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/2807/steps/12/logs/stdio
Comment 2 Randy MacLeod 2025-11-27 15:52:17 UTC
Probably "fix" by masking the warning... probably.
In theory we could deadlock but we've been doing this for years and 
haven't had a real issue appear.
Comment 3 Mathieu Dubois-Briand 2025-12-02 09:23:53 UTC
qemuarm64-ltp debian13-vk-arm1 mathieu/master-next completed at 2025-11-29 11:05:31+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/98/builds/2655/steps/11/logs/stdio
Comment 4 Mathieu Dubois-Briand 2025-12-03 12:48:28 UTC
qemuarm64-armhost debian13-vk-arm1 master completed at 2025-12-03 01:42:23+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/2832/steps/12/logs/stdio

qemux86 ubuntu2404-vk-3 mathieu/master-next completed at 2025-12-03 10:07:58+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/2788/steps/14/logs/stdio
Comment 5 Mathieu Dubois-Briand 2025-12-04 09:32:42 UTC
qemuarm64-armhost debian13-vk-arm1 master completed at 2025-12-03 21:55:57+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/2837/steps/12/logs/stdio
Comment 6 Mathieu Dubois-Briand 2025-12-04 12:22:43 UTC
qemuarm64-ltp debian13-vk-arm1 mathieu/master-next completed at 2025-12-04 11:30:49+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/98/builds/2672/steps/11/logs/stdio

qemux86-world opensuse156-vk-1 mathieu/master-next completed at 2025-12-04 10:47:38+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/2825/steps/12/logs/stdio
Comment 7 Mathieu Dubois-Briand 2025-12-05 12:09:04 UTC
I was not able to reproduce this locally, but I believe the filter we have in bitbake-server and bitbake-worker should be imported to bitbake-layers.

I've created such a patch, I will let it sit in my branch for a few days to check if the issue is indeed fixed.

https://git.openembedded.org/bitbake-contrib/commit/?h=mathieu/master-next&id=ba07270ef9fbdaeaf43f4ef18d71098e8e4cb11e
Comment 8 Mathieu Dubois-Briand 2025-12-08 10:18:01 UTC
qemuarm64-ptest debian13-vk-arm1 master completed at 2025-12-08 01:37:13+00:00
https://valkyrie.yocto.io/pub/non-release/20251208-8/testresults/qemuarm64-ptest/
https://autobuilder.yoctoproject.org/valkyrie/#/builders/61/builds/2682/steps/11/logs/stdio
Comment 10 Mathieu Dubois-Briand 2025-12-15 13:01:54 UTC
Reopen: I did miss that some occurrences were not from bitbake-config, but from the main bitbake script. Which is a bit strange IMO: why are we only seeing these kind of issues now?
Comment 11 Mathieu Dubois-Briand 2025-12-15 13:11:35 UTC
qemux86-64-x32 rocky8-vk-1 mathieu/master-next completed at 2025-12-13 09:20:41+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/11/builds/2820/steps/14/logs/stdio
Comment 12 Mathieu Dubois-Briand 2025-12-19 08:14:04 UTC
Sent a new fix: https://lists.openembedded.org/g/bitbake-devel/message/18612
Comment 13 Mathieu Dubois-Briand 2026-01-05 09:48:28 UTC
qemux86-64-ltp ubuntu2404-vk-1 master completed at 2026-01-04 21:41:24+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2792/steps/12/logs/stdio

SERIOUSLY!!!
Comment 14 Chen Qi 2026-01-05 10:16:34 UTC
What if we put the filter in lib/bb/__init__.py? Does that work?
If that work, then bin/bitbake-xxx programs will not need that same filter?
Comment 15 Mathieu Dubois-Briand 2026-01-05 10:35:58 UTC
(In reply to Chen Qi from comment #14)
> What if we put the filter in lib/bb/__init__.py? Does that work?
> If that work, then bin/bitbake-xxx programs will not need that same filter?

Looking at the documentation, this should work. I will try modify the code in that way.
Comment 16 Mathieu Dubois-Briand 2026-01-26 14:07:08 UTC
Importing failures logged to the wrong bugzilla entry. All of these come from the "bitbake" script and should be fixed by the patch present in my branch.

qemuarm64-armhost debian13-vk-arm1 master completed at 2026-01-12 08:46:57+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/3033/steps/13/logs/stdio

qemux86-64-ltp rocky8-vk-1 master-next completed at 2026-01-13 14:27:32+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/63/builds/2840/steps/13/logs/stdio

qemuarm-oecore rocky8-vk-1 master-next completed at 2026-01-13 19:59:08+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3005/steps/13/logs/stdio

qemux86-alt rocky8-vk-1 master&master-next completed at 2026-01-14 18:27:43+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/3001/steps/15/logs/stdio

qa-extras2 alma8-vk-2 master completed at 2026-01-14 20:43:31+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/3011/steps/13/logs/stdio

qemux86-world-alt alma8-vk-1 master-next&master completed at 2026-01-20 14:55:49+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/2903/steps/13/logs/stdio
Comment 17 Mathieu Dubois-Briand 2026-01-31 09:25:14 UTC
qemuarm64 debian13-vk-2 master completed at 2026-01-31 02:28:11+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3120/steps/14/logs/stdio

qemux86-world ubuntu2504-vk-1 master completed at 2026-01-31 01:21:36+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3121/steps/12/logs/stdio
Comment 18 Mathieu Dubois-Briand 2026-02-04 12:04:36 UTC
wic opensuse155-vk-2 master completed at 2026-02-03 01:26:44+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/15/builds/3075/steps/13/logs/stdio