We got this twice I think: once on mathieu's master-next and once on master: ERROR: build-sysroots-1.0-r0 do_build_target_sysroot: Error executing a python function in exec_func_python() autogenerated: 1318 1319 The stack trace of python calls that resulted in this exception/failure was: 1320 File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 1321 0001: 1322 *** 0002:do_build_target_sysroot(d) 1323 0003: 1324 File: '/tmp/devtoolqa5qzujc07/core-copy/meta/recipes-core/meta/build-sysroots.bb', lineno: 46, function: do_build_target_sysroot 1325 0042: targetsysroot = d.getVar("STANDALONE_SYSROOT") 1326 0043: nativesysroot = d.getVar("STANDALONE_SYSROOT_NATIVE") 1327 0044: import os 1328 0045: os.environ['PATH'] = "%s/bin:%s/usr/bin:%s" % (nativesysroot, nativesysroot, os.environ['PATH']) 1329 *** 0046: staging_populate_sysroot_dir(targetsysroot, nativesysroot, False, d) 1330 0047:} 1331 0048:do_build_target_sysroot[cleandirs] = "${STANDALONE_SYSROOT}" 1332 0049:do_build_target_sysroot[nostamp] = "1" 1333 0050:addtask do_build_target_sysroot 1334 File: '/tmp/devtoolqa5qzujc07/core-copy/meta/classes-global/staging.bbclass', lineno: 249, function: staging_populate_sysroot_dir 1335 0245: continue 1336 0246: 1337 0247: staging_processfixme(fixme, targetdir, targetsysroot, nativesysroot, d) 1338 0248: for p in sorted(postinsts): 1339 *** 0249: bb.note("Running postinst {}, output:\n{}".format(p, subprocess.check_output(p, shell=True, stderr=subprocess.STDOUT))) 1340 0250: 1341 0251:# 1342 0252:# Manifests here are complicated. The main sysroot area has the unpacked sstate 1343 0253:# which us unrelocated and tracked by the main sstate manifests. Each recipe 1344 File: '/usr/lib/python3.9/subprocess.py', lineno: 424, function: check_output 1345 0420: else: 1346 0421: empty = b'' 1347 0422: kwargs['input'] = empty 1348 0423: 1349 *** 0424: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, 1350 0425: **kwargs).stdout 1351 0426: 1352 0427: 1353 0428:class CompletedProcess(object): 1354 File: '/usr/lib/python3.9/subprocess.py', lineno: 528, function: run 1355 0524: # We don't call process.wait() as .__exit__ does that for us. 1356 0525: raise 1357 0526: retcode = process.poll() 1358 0527: if check and retcode: 1359 *** 0528: raise CalledProcessError(retcode, process.args, 1360 0529: output=stdout, stderr=stderr) 1361 0530: return CompletedProcess(process.args, retcode, stdout, stderr) 1362 0531: 1363 0532: 1364 Exception: subprocess.CalledProcessError: Command '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3470738/tmp/sysroots/qemux86-64/usr/bin/postinst-useradd-02user-openssh' returned non-zero exit status 1. 1365 1366 Subprocess output: 1367 /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3470738/tmp/sysroots/x86_64/usr/sbin/useradd 1368 Running useradd commands... 1369 NOTE: openssh: Performing useradd with [--root /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3470738/tmp/sysroots/qemux86-64 --system --no-create-home --home-dir /var/run/sshd --shell /bin/false --user-group sshd] 1370 awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory 1371 useradd: Warning: missing or non-executable shell '/bin/false' 1372 useradd: user 'sshd' already exists 1373 ERROR: openssh: useradd command did not succeed.
oe-selftest-debian debian11-vk-1 master completed at 2026-01-14 20:57:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3019/steps/15/logs/stdio oe-selftest-fedora alma8-vk-2 mathieu/master-next completed at 2026-01-12 10:28:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2889/steps/16/logs/stdio
I've tried to reproduce this issue in multiple ways, building on multiple Debian versions, but haven't been able to trigger the error. So I'm investigating based on the failed master build which Antonin linked to (https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3019/steps/15/logs/stdio). As this is a master build not master-next, it's not related to the devtool path changes I submitted as those have not been merged to master yet. I've also reviewed the failing test case and the relevant bits of devtool, it's not hitting the lines of code I modified in devtool. The failing command is `bitbake build-sysroots:do_build_target_sysroot build-sysroots:do_build_native_sysroot`, called by the command `devtool ide-sdk -m shared oe-selftest-image cmake-example meson-example --ide=code`, called by the test case `devtool.DevtoolIdeSdkTests.test_devtool_ide_sdk_shared_sysroots`. In `do_build_target_sysroot()`, we call `staging_populate_sysroot_dir()` which runs postinst scripts for the packages in the sysroot. The failing postinst script is `postinst-useradd-02user-openssh`. The line `/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3470738/tmp/sysroots/x86_64/usr/sbin/useradd` is printed by `command -v useradd` in `user_group_groupmems_add_sysroot()`. This is a path into the native sysroot so we aren't running a target binary here. In `perform_useradd()`, we use awk to parse out the username to be added so we can check if it already exists. Execution of awk fails with the error message `awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory`. This results in the `username` variable being an empty string, so `user_exists` is false regardless of whether the user already exists or not. So, we get the error `useradd: user 'sshd' already exists` because we're unnecessarily running useradd for a user we've already created. The issue to fix is awk failing. What's strange is that no awk that I can find links against libncurses, of any version. The only binaries in the native sysroot which link against libncurses5 (after running `do_build_target_sysroot()`) are ccmake, cfdisk & irqtop. Debian 11 uses libncurses6 (not 5), so the failing command must be something we've built. However, gawk-native is in ASSUME_PROVIDED so this comes from the host, it's not something we have built. I am thoroughly confused. So, I've re-run the failing build on the autobuilder to see if this happens again (https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3059). If I can inspect the binaries in the failing case then hopefully it'll be clearer what is going wrong.
Paul, I'm hitting a different issue, although somewhat very similar, but that only seems to happen when applying Adrian's series[1] (with patch 13/14 dropped as Adrian told me to do). It's not the same error, as I don't have "awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory". But what might help you is that I've pushed a yocto-autobuilder-helper branch that executes _only_ this test[2]. This allowed me to reproduce my error in 5 minutes[3]. Hope that helps. [1]: https://lore.kernel.org/r/20251231114718.4031606-1-adrian.freihofer@siemens.com [2]: https://git.yoctoproject.org/yocto-autobuilder-helper/log/?h=contrib/agodard/oe-selftest-ide-sdk [3]: autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2955
This time this groupadd command failed, but still there is awk not loading a shared library. This time, however, I did have Adrian's series in my branch (https://lore.kernel.org/r/20251231114718.4031606-1-adrian.freihofer@siemens.com). But since this error already happened without Adrian's series, I'm still going to consider it AB-INT here. oe-selftest-debian debian12-vk-5 agodard/master-next completed at 2026-01-16 16:11:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3035/steps/15/logs/stdio
Rare-ish, difficult to reproduce failure. Move to M2.
Same discussion is here: https://lists.openembedded.org/g/openembedded-core/topic/117464897#msg229962 It reproduces this issue quite often. And I also proposed a patch, which was shot down immediately. But I think it could be a solution.
I have to admit that it looks much stranger than I thought three weeks ago when I first saw it. At the moment, I can't reproduce it either.
I'm able to reproduce it in a minute now. - Today's poky master - with this patch https://patchwork.yoctoproject.org/project/oe-core/patch/20260126073809.468495-5-adrian.freihofer@siemens.com/. The patch adds usedadd to the cmake-example recipe. I guess the issue could be reproduces with other recipes which add users, without this patch, but I did not try. It is also in oe-core-contrib on branch https://git.openembedded.org/openembedded-core-contrib/log/?h=adrianf/ide-sdk-improvements - devtool ide-sdk cmake-example - rm tmp - devtool ide-sdk cmake-example Exception: subprocess.CalledProcessError: Command '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64/usr/bin/postinst-base-passwd' returned non-zero exit status 1. Subprocess output: /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/x86_64/usr/sbin/useradd Running groupadd commands... NOTE: cmake-example: Performing groupadd with [--root /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64 --system cmake-example] awk: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory ERROR: cmake-example: groupadd command did not succeed. This seems to happen always on my setup when re-building with clean tmp. I tried that multiple times. The recipe is not rebuilt, the re-build fails with "awk: error ..." Then I tried to print the PATH variable where the "Running groupadd commands..." is printed. Searching for this string leads into useradd.bbclass https://git.openembedded.org/openembedded-core/tree/meta/classes/useradd.bbclass#n57. Changing the string does not have any impact. The package is not re-compiled. The re-build fails with "awk: error ..." I was wondering if bitbake parses my changes in the code at all. Yes, it does: bitbake-getvar -r cmake-example useradd_preinst # # $useradd_preinst [4 operations] # set ast.py:200 [eval] # [func] "1" ... # # Perform group additions first, since user additions may depend # # on these groups existing # if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then # exit 1 # echo "Running groupadd commands... foo" # which awk # echo "PATH: $PATH" Very unexpected: After calling bitbake-getvar, the rebuild with clean tmp works. (Not sure how it "works" with the exit 1) Reverting the changes in the useradd_preinst, cleaning the tmp and re-building brings me back to the "awk: error ..." I do not yet believe that yet and I hope I'm not going to confuse others, but the mess which I'm describing here, looks like fully reproducible on my machine. Another detail is that this line https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/meta/build-sysroots.bb#n34 sets the PATH to sysroot only. Not sure if this kills the ASSUME_PROVIDED concept be removing the host path? Just for clarification, since this all seams to happen in the context of devtool ide-sdk. The design of devtool ide-sdk is aiming for working around all these kind of issues by using the per recipe sysroots for all development workflows. The default mode does not need the shared sysroots and does not trigger this issue. Only if devtool ide-sdk is called with --mode=shared, it creates a SDK with a shared sysroot which requires all this special code paths. The --mode=shared is only supported because I know that some users like it. However, the design goal of ide-sdk's default mode is to make the shared sysroot mode completely unneeded because all develpoment workflows are straight forward with regular per recipe sysroot builds.
Root cause analysis: I guess it is related to this issue https://bugzilla.yoctoproject.org/show_bug.cgi?id=16072 and the related fix: https://git.openembedded.org/openembedded-core/commit/meta?id=c5bbf0a60b1d63e68f849a63e5d3872954e7cd3f devtool -d ide-sdk -m shared cmake-example does: bitbake meta-ide-support:do_build cmake-example:do_populate_sysroot bitbake build-sysroots:do_build_target_sysroot \ build-sysroots:do_build_native_sysroot bitbake meta-ide-support -g shows: "glibc.do_prepare_recipe_sysroot" -> "gawk-native.do_populate_sysroot" glibc.inc: DEPENDS = ... gawk-replacement-native" bitbake.conf: ASSUME_PROVIDED = "... gawk-native... gawk.bb PROVIDES:append:class-native = " gawk-replacement-native" This brings us to a situation where we use awk from the host but when build-sysroots:do_build_native_sysroot runs the awk-native appears in the path. If awk gets populated before the libraries, the error happens. That probably means that we have to use awk-native everywhere and remove it entirely from ASSUME_PROVIDED.
Thanks for the analysis Adrian. I'm afraid I still don't quite understand the race though. build-sysroots:do_build_native_sysroot calls: staging_populate_sysroot_dir(targetsysroot, nativesysroot, True, d) which basically does a: for pkgarch in pkgarchs: for manifest in glob.glob(d.expand("${SSTATE_MANIFESTS}/manifest-%s-*.populate_sysroot" % pkgarch)): <install files> staging_processfixme(...) for p in sorted(postinsts): bb.note("Running postinst {}") i.e. all the libs are installed, *then* the postinsts are run. The libraries would be present before the binaries were run. I did have a theory though, I wonder if the two tasks race. You say we run: bitbake build-sysroots:do_build_target_sysroot \ build-sysroots:do_build_native_sysroot and maybe this needs to be: bitbake build-sysroots:do_build_target_sysroot; \ bitbake build-sysroots:do_build_native_sysroot i.e. run then serially?
I didn't explicitly write that, but yes, my theory is that do_build_target_sysroot uses awk, while do_build_native_sysroot installs gawk-native in the shared sysroot which is in the PATH of do_build_target_sysroot. Since the installation of gawk-native is not atomic (libraries, awk), there is a potential race problem: do_build_target_sysroot calls a partially installed awk, which we can observe. Yes, we could serialize these two tasks manually. I was also thinking about replacing bitbake build-sysroots:do_build_target_sysroot \ build-sysroots:do_build_native_sysroot by bitbake build-sysroots:do_build_target_sysroot bitbake build-sysroots:do_build_native_sysroot However, I am not entirely convinced that this is the right solution to this problem. At least in theory, Bitbake should know the dependencies between tasks and isolate things with sysroots per recipe. Obviously, both concepts are not fully applicable to this use case with shared sysroots. The question is: Where will we have the least negative consequences if we abandon fundamental concepts? I also see another approach to serializing the two tasks manually: We could blame https://git.openembedded.org/openembedded-core/commit/meta?id=c5bbf0a60b1d63e68f849a63e5d3872954e7cd3f. Could we revert it and solve the reproducibility issue by adding awk to uninative? I also believe that commit c5bbf0a60b1d63e68f849a63e5d3872954e7cd3f introduces significant complexity, the consequences of which are difficult to predict, as it introduces a mixture of awk in ASSUME_PROVIDED and awk-native. Probably I should send a patch which serializes the tasks, at least as a next step.
Here is a branch which does what Richard suggested in today's patch meeting: https://git.openembedded.org/openembedded-core-contrib/log/?h=adrianf/bug-16135 I will test and send it tomorrow.
Tested successfully. Added some prints to both functions build-sysroots:do_build_target_sysroot and build-sysroots:do_build_native_sysroot. With locks it looks like that: WARNING: build-sysroots-1.0-r0 do_build_target_sysroot: target sysroot build started WARNING: build-sysroots-1.0-r0 do_build_target_sysroot: target sysroot build ended WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build started WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build ended NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded. Without locks I can see this very often: WARNING: build-sysroots-1.0-r0 do_build_target_sysroot: target sysroot build started WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build started ERROR: build-sysroots-1.0-r0 do_build_target_sysroot: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_build_target_sysroot(d) 0003: File: '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/../../layers/openembedded-core/meta/recipes-core/meta/build-sysroots.bb', lineno: 42, function: do_build_target_sysroot 0038: targetsysroot = d.getVar("STANDALONE_SYSROOT") 0039: nativesysroot = d.getVar("STANDALONE_SYSROOT_NATIVE") 0040: import os 0041: os.environ['PATH'] = "%s/bin:%s/usr/bin:%s" % (nativesysroot, nativesysroot, os.environ['PATH']) *** 0042: staging_populate_sysroot_dir(targetsysroot, nativesysroot, False, d) 0043: bb.warn("target sysroot build ended") 0044:} 0045:do_build_target_sysroot[cleandirs] = "${STANDALONE_SYSROOT}" 0046:do_build_target_sysroot[nostamp] = "1" File: '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/../../layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 249, function: staging_populate_sysroot_dir 0245: continue 0246: 0247: staging_processfixme(fixme, targetdir, targetsysroot, nativesysroot, d) 0248: for p in sorted(postinsts): *** 0249: bb.note("Running postinst {}, output:\n{}".format(p, subprocess.check_output(p, shell=True, stderr=subprocess.STDOUT))) 0250: 0251:# 0252:# Manifests here are complicated. The main sysroot area has the unpacked sstate 0253:# which us unrelocated and tracked by the main sstate manifests. Each recipe File: '/usr/lib64/python3.13/subprocess.py', lineno: 472, function: check_output 0468: else: 0469: empty = b'' 0470: kwargs['input'] = empty 0471: *** 0472: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, 0473: **kwargs).stdout 0474: 0475: 0476:class CompletedProcess(object): File: '/usr/lib64/python3.13/subprocess.py', lineno: 577, function: run 0573: # We don't call process.wait() as .__exit__ does that for us. 0574: raise 0575: retcode = process.poll() 0576: if check and retcode: *** 0577: raise CalledProcessError(retcode, process.args, 0578: output=stdout, stderr=stderr) 0579: return CompletedProcess(process.args, retcode, stdout, stderr) 0580: 0581: Exception: subprocess.CalledProcessError: Command '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64/usr/bin/postinst-base-passwd' returned non-zero exit status 1. Subprocess output: /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/x86_64/usr/sbin/useradd Running groupadd commands... NOTE: cmake-example: Performing groupadd with [--root /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64 --system cmake-example] awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory ERROR: cmake-example: groupadd command did not succeed. ERROR: Logfile of failure stored in: /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/work/qemux86_64-poky-linux/build-sysroots/1.0/temp/log.do_build_target_sysroot.925535 ERROR: Task (/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/../../layers/openembedded-core/meta/recipes-core/meta/build-sysroots.bb:do_build_target_sysroot) failed with exit code '1' WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build ended NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and 1 failed. Fix is here: - https://lists.openembedded.org/g/openembedded-core/message/230185 - https://patchwork.yoctoproject.org/project/oe-core/list/?series=42835
Thanks Adrian! https://git.openembedded.org/openembedded-core/commit/?id=acc84167443026012ffc5a79bbd907be72b6a03c