Bug 16135

oe-selftest-debian debian11-vk-1 master completed at 2026-01-14 20:57:12+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3019/steps/15/logs/stdio

oe-selftest-fedora alma8-vk-2 mathieu/master-next completed at 2026-01-12 10:28:10+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2889/steps/16/logs/stdio

I've tried to reproduce this issue in multiple ways, building on multiple Debian versions, but haven't been able to trigger the error. So I'm investigating based on the failed master build which Antonin linked to (https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3019/steps/15/logs/stdio).

As this is a master build not master-next, it's not related to the devtool path changes I submitted as those have not been merged to master yet. I've also reviewed the failing test case and the relevant bits of devtool, it's not hitting the lines of code I modified in devtool.

The failing command is `bitbake build-sysroots:do_build_target_sysroot build-sysroots:do_build_native_sysroot`, called by the command `devtool ide-sdk -m shared oe-selftest-image cmake-example meson-example --ide=code`, called by the test case `devtool.DevtoolIdeSdkTests.test_devtool_ide_sdk_shared_sysroots`.

In `do_build_target_sysroot()`, we call `staging_populate_sysroot_dir()` which runs postinst scripts for the packages in the sysroot. The failing postinst script is `postinst-useradd-02user-openssh`.

The line `/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3470738/tmp/sysroots/x86_64/usr/sbin/useradd` is printed by `command -v useradd` in `user_group_groupmems_add_sysroot()`. This is a path into the native sysroot so we aren't running a target binary here.

In `perform_useradd()`, we use awk to parse out the username to be added so we can check if it already exists. Execution of awk fails with the error message `awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory`. This results in the `username` variable being an empty string, so `user_exists` is false regardless of whether the user already exists or not. So, we get the error `useradd: user 'sshd' already exists` because we're unnecessarily running useradd for a user we've already created.

The issue to fix is awk failing. What's strange is that no awk that I can find links against libncurses, of any version. The only binaries in the native sysroot which link against libncurses5 (after running `do_build_target_sysroot()`) are ccmake, cfdisk & irqtop. Debian 11 uses libncurses6 (not 5), so the failing command must be something we've built. However, gawk-native is in ASSUME_PROVIDED so this comes from the host, it's not something we have built.

I am thoroughly confused. So, I've re-run the failing build on the autobuilder to see if this happens again (https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3059). If I can inspect the binaries in the failing case then hopefully it'll be clearer what is going wrong.

Paul, I'm hitting a different issue, although somewhat very similar, but that only seems to happen when applying Adrian's series[1] (with patch 13/14 dropped as Adrian told me to do).

It's not the same error, as I don't have "awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory".

But what might help you is that I've pushed a yocto-autobuilder-helper branch that executes _only_ this test[2]. This allowed me to reproduce my error in 5 minutes[3].

Hope that helps.

[1]: https://lore.kernel.org/r/20251231114718.4031606-1-adrian.freihofer@siemens.com
[2]: https://git.yoctoproject.org/yocto-autobuilder-helper/log/?h=contrib/agodard/oe-selftest-ide-sdk
[3]: autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2955

This time this groupadd command failed, but still there is awk not loading a
shared library. This time, however, I did have Adrian's series in my branch
(https://lore.kernel.org/r/20251231114718.4031606-1-adrian.freihofer@siemens.com).

But since this error already happened without Adrian's series, I'm still going
to consider it AB-INT here.

oe-selftest-debian debian12-vk-5 agodard/master-next completed at 2026-01-16 16:11:37+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3035/steps/15/logs/stdio

Rare-ish, difficult to reproduce failure. Move to M2.

Same discussion is here: https://lists.openembedded.org/g/openembedded-core/topic/117464897#msg229962

It reproduces this issue quite often.

And I also proposed a patch, which was shot down immediately. But I think it could be a solution.

I have to admit that it looks much stranger than I thought three weeks ago when I first saw it. At the moment, I can't reproduce it either.

I'm able to reproduce it in a minute now.

- Today's poky master
- with this patch https://patchwork.yoctoproject.org/project/oe-core/patch/20260126073809.468495-5-adrian.freihofer@siemens.com/. The patch adds usedadd to the cmake-example recipe. I guess the issue could be reproduces with other recipes which add users, without this patch, but I did not try. It is also in oe-core-contrib on branch https://git.openembedded.org/openembedded-core-contrib/log/?h=adrianf/ide-sdk-improvements
- devtool ide-sdk cmake-example
- rm tmp
- devtool ide-sdk cmake-example

Exception: subprocess.CalledProcessError: Command '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64/usr/bin/postinst-base-passwd' returned non-zero exit status 1.
                                                                                                          
Subprocess output:   
/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/x86_64/usr/sbin/useradd
Running groupadd commands...
NOTE: cmake-example: Performing groupadd with [--root /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64 --system cmake-example]
awk: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
ERROR: cmake-example: groupadd command did not succeed.

This seems to happen always on my setup when re-building with clean tmp. I tried that multiple times. The recipe is not rebuilt, the re-build fails with "awk: error ..."

Then I tried to print the PATH variable where the "Running groupadd commands..." is printed. Searching for this string leads into useradd.bbclass https://git.openembedded.org/openembedded-core/tree/meta/classes/useradd.bbclass#n57. Changing the string does not have any impact. The package is not re-compiled. The re-build fails with "awk: error ..."

I was wondering if bitbake parses my changes in the code at all. Yes, it does:

bitbake-getvar -r cmake-example useradd_preinst
#
# $useradd_preinst [4 operations]
#   set ast.py:200 [eval]
#     [func] "1"
...
#     # Perform group additions first, since user additions may depend
#     # on these groups existing
#     if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then
#       exit 1
#       echo "Running groupadd commands... foo"
#       which awk
#       echo "PATH: $PATH"

Very unexpected: After calling bitbake-getvar, the rebuild with clean tmp works. (Not sure how it "works" with the exit 1)

Reverting the changes in the useradd_preinst, cleaning the tmp and re-building brings me back to the "awk: error ..."

I do not yet believe that yet and I hope I'm not going to confuse others, but the mess which I'm describing here, looks like fully reproducible on my machine.

Another detail is that this line https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/meta/build-sysroots.bb#n34 sets the PATH to sysroot only. Not sure if this kills the ASSUME_PROVIDED concept be removing the host path?

Just for clarification, since this all seams to happen in the context of devtool ide-sdk. The design of devtool ide-sdk is aiming for working around all these kind of issues by using the per recipe sysroots for all development workflows. The default mode does not need the shared sysroots and does not trigger this issue. Only if devtool ide-sdk is called with --mode=shared, it creates a SDK with a shared sysroot which requires all this special code paths. The --mode=shared is only supported because I know that some users like it. However, the design goal of ide-sdk's default mode is to make the shared sysroot mode completely unneeded because all develpoment workflows are straight forward with regular per recipe sysroot builds.

Root cause analysis:

I guess it is related to this issue https://bugzilla.yoctoproject.org/show_bug.cgi?id=16072 and the related fix: https://git.openembedded.org/openembedded-core/commit/meta?id=c5bbf0a60b1d63e68f849a63e5d3872954e7cd3f

devtool -d ide-sdk -m shared cmake-example does:

  bitbake meta-ide-support:do_build cmake-example:do_populate_sysroot
  bitbake build-sysroots:do_build_target_sysroot \
          build-sysroots:do_build_native_sysroot


bitbake meta-ide-support -g shows:
"glibc.do_prepare_recipe_sysroot" -> "gawk-native.do_populate_sysroot"

glibc.inc:
DEPENDS = ... gawk-replacement-native"

bitbake.conf:
ASSUME_PROVIDED = "... gawk-native...

gawk.bb
PROVIDES:append:class-native = " gawk-replacement-native"

This brings us to a situation where we use awk from the host but when build-sysroots:do_build_native_sysroot runs the awk-native appears in the path. If awk gets populated before the libraries, the error happens.

That probably means that we have to use awk-native everywhere and remove it entirely from ASSUME_PROVIDED.

Thanks for the analysis Adrian. I'm afraid I still don't quite understand the race though.

build-sysroots:do_build_native_sysroot calls:

    staging_populate_sysroot_dir(targetsysroot, nativesysroot, True, d)

which basically does a:

    for pkgarch in pkgarchs:
        for manifest in glob.glob(d.expand("${SSTATE_MANIFESTS}/manifest-%s-*.populate_sysroot" % pkgarch)):
            <install files>
    staging_processfixme(...)
    for p in sorted(postinsts):
        bb.note("Running postinst {}")

i.e. all the libs are installed, *then* the postinsts are run. The libraries would be present before the binaries were run.

I did have a theory though, I wonder if the two tasks race. You say we run:

  bitbake build-sysroots:do_build_target_sysroot \
          build-sysroots:do_build_native_sysroot

and maybe this needs to be:

  bitbake build-sysroots:do_build_target_sysroot; \
  bitbake build-sysroots:do_build_native_sysroot

i.e. run then serially?

I didn't explicitly write that, but yes, my theory is that do_build_target_sysroot uses awk, while do_build_native_sysroot installs gawk-native in the shared sysroot which is in the PATH of do_build_target_sysroot. Since the installation of gawk-native is not atomic (libraries, awk), there is a potential race problem: do_build_target_sysroot calls a partially installed awk, which we can observe.

Yes, we could serialize these two tasks manually. I was also thinking about replacing
  bitbake build-sysroots:do_build_target_sysroot \
          build-sysroots:do_build_native_sysroot
by
  bitbake build-sysroots:do_build_target_sysroot
  bitbake build-sysroots:do_build_native_sysroot

However, I am not entirely convinced that this is the right solution to this problem. At least in theory, Bitbake should know the dependencies between tasks and isolate things with sysroots per recipe. Obviously, both concepts are not fully applicable to this use case with shared sysroots. The question is: Where will we have the least negative consequences if we abandon fundamental concepts?

I also see another approach to serializing the two tasks manually: We could blame https://git.openembedded.org/openembedded-core/commit/meta?id=c5bbf0a60b1d63e68f849a63e5d3872954e7cd3f. Could we revert it and solve the reproducibility issue by adding awk to uninative?

I also believe that commit c5bbf0a60b1d63e68f849a63e5d3872954e7cd3f introduces significant complexity, the consequences of which are difficult to predict, as it introduces a mixture of awk in ASSUME_PROVIDED and awk-native. 

Probably I should send a patch which serializes the tasks, at least as a next step.

Here is a branch which does what Richard suggested in today's patch meeting: https://git.openembedded.org/openembedded-core-contrib/log/?h=adrianf/bug-16135

I will test and send it tomorrow.

Tested successfully.

Added some prints to both functions build-sysroots:do_build_target_sysroot and   build-sysroots:do_build_native_sysroot.

With locks it looks like that:

WARNING: build-sysroots-1.0-r0 do_build_target_sysroot: target sysroot build started
WARNING: build-sysroots-1.0-r0 do_build_target_sysroot: target sysroot build ended
WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build started
WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build ended
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.


Without locks I can see this very often:

WARNING: build-sysroots-1.0-r0 do_build_target_sysroot: target sysroot build started                                                                                                                                
WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build started                                                                                                                                
ERROR: build-sysroots-1.0-r0 do_build_target_sysroot: Error executing a python function in exec_func_python() autogenerated:                              
                                                                                                                                                                                                                    
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:                                                                                              
 *** 0002:do_build_target_sysroot(d)   
     0003:                                                                                        
File: '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/../../layers/openembedded-core/meta/recipes-core/meta/build-sysroots.bb', lineno: 42, function: do_build_target_sysroot                                                                                                                                                                                                                                              
     0038:    targetsysroot = d.getVar("STANDALONE_SYSROOT")
     0039:    nativesysroot = d.getVar("STANDALONE_SYSROOT_NATIVE")                                                                                                                                                                                                                                                                                                                                                                      
     0040:    import os                                                                                  
     0041:    os.environ['PATH'] = "%s/bin:%s/usr/bin:%s" % (nativesysroot, nativesysroot, os.environ['PATH'])
 *** 0042:    staging_populate_sysroot_dir(targetsysroot, nativesysroot, False, d)            
     0043:    bb.warn("target sysroot build ended")
     0044:}                                                                                                                                                                                                                                                                                                                                                                                                                              
     0045:do_build_target_sysroot[cleandirs] = "${STANDALONE_SYSROOT}"
     0046:do_build_target_sysroot[nostamp] = "1"   
File: '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/../../layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 249, function: staging_populate_sysroot_dir
     0245:                        continue
     0246:                     
     0247:    staging_processfixme(fixme, targetdir, targetsysroot, nativesysroot, d)
     0248:    for p in sorted(postinsts):
 *** 0249:        bb.note("Running postinst {}, output:\n{}".format(p, subprocess.check_output(p, shell=True, stderr=subprocess.STDOUT)))
     0250:                         
     0251:#                  
     0252:# Manifests here are complicated. The main sysroot area has the unpacked sstate
     0253:# which us unrelocated and tracked by the main sstate manifests. Each recipe
File: '/usr/lib64/python3.13/subprocess.py', lineno: 472, function: check_output
     0468:        else:
     0469:            empty = b''                                                  
     0470:        kwargs['input'] = empty
     0471:                                                              
 *** 0472:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
     0473:               **kwargs).stdout
     0474:                                                                                                                                                                                                                                                                                                                                                                                                                               
     0475:                                                                                                                                                                                                                                                                                                                                                                                                                               
     0476:class CompletedProcess(object):
File: '/usr/lib64/python3.13/subprocess.py', lineno: 577, function: run                           
     0573:            # We don't call process.wait() as .__exit__ does that for us.                                                                                                                                 
     0574:            raise            
     0575:        retcode = process.poll()                
     0576:        if check and retcode:                                                                                                                                                                                                                                                                                                                                                                                                  
 *** 0577:            raise CalledProcessError(retcode, process.args,
     0578:                                     output=stdout, stderr=stderr)
     0579:    return CompletedProcess(process.args, retcode, stdout, stderr)
     0580:          
     0581:                     
Exception: subprocess.CalledProcessError: Command '/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64/usr/bin/postinst-base-passwd' returned non-zero exit status 1.
                                                                                                          
Subprocess output:                        
/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/x86_64/usr/sbin/useradd
Running groupadd commands... 
NOTE: cmake-example: Performing groupadd with [--root /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/sysroots/qemux86-64 --system cmake-example]
awk: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory
ERROR: cmake-example: groupadd command did not succeed.
                                                                                                          
ERROR: Logfile of failure stored in: /home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/tmp/work/qemux86_64-poky-linux/build-sysroots/1.0/temp/log.do_build_target_sysroot.925535
ERROR: Task (/home/adrian/projets/oss/oe/poky-master-sm/build/poky-master/../../layers/openembedded-core/meta/recipes-core/meta/build-sysroots.bb:do_build_target_sysroot) failed with exit code '1'
WARNING: build-sysroots-1.0-r0 do_build_native_sysroot: native sysroot build ended
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and 1 failed.

Fix is here:
- https://lists.openembedded.org/g/openembedded-core/message/230185
- https://patchwork.yoctoproject.org/project/oe-core/list/?series=42835

Thanks Adrian!

https://git.openembedded.org/openembedded-core/commit/?id=acc84167443026012ffc5a79bbd907be72b6a03c