Bug 13904

Summary: do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Jackie Huang <jackie.huang>
Component: oe-core otherAssignee: Eilís Ní Fhlannagáin <pidge>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: ernstp, frederic.martinsons, louis.rannou, ludovic.jozeau+yocto-bug, Martin.Jansa, p.lobacz, pidge, randy.macleod, steve
Version: 2.7.4   
Target Milestone: 5.0 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Jackie Huang 2020-05-18 21:10:53 UTC
This is found in "warrior", but I think it may also exists in later releases.

Observed error:
=====================
ERROR: stx-metal-1.0.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/ws-stx-i64_20200518/src_layers/poky/meta/classes/staging.bbclass', lineno: 566, function: extend_recipe_sysroot
     0562:    for f in fixme:
     0563:        staging_processfixme(fixme[f], f, recipesysroot, recipesysrootnative, d)
     0564:
     0565:    for p in postinsts:
 *** 0566:        subprocess.check_output(p, shell=True, stderr=subprocess.STDOUT)
     0567:
     0568:    for dep in manifests:
     0569:        c = setscenedeps[dep][0]
     0570:        os.symlink(manifests[dep], depdir + "/" + c + ".complete")
File: '/usr/lib/python3.6/subprocess.py', lineno: 356, function: check_output
     0352:        # empty string. That is maintained here for backwards compatibility.
     0353:        kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
     0354:
     0355:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 *** 0356:               **kwargs).stdout
     0357:
     0358:
     0359:class CompletedProcess(object):
     0360:    """A process that has finished running.
File: '/usr/lib/python3.6/subprocess.py', lineno: 438, function: run
     0434:            raise
     0435:        retcode = process.poll()
     0436:        if check and retcode:
     0437:            raise CalledProcessError(retcode, process.args,
 *** 0438:                                     output=stdout, stderr=stderr)
     0439:    return CompletedProcess(process.args, retcode, stdout, stderr)
     0440:
     0441:
     0442:def list2cmdline(seq):
Exception: subprocess.CalledProcessError: Command '/ws-stx-i64_20200518/prj_stx/tmp/work/corei7-64-poky-linux/stx-metal/1.0.0-r0/recipe-sysroot/usr/bin/postinst-useradd-fm-mgr' returned non-zero exit status 1.

Subprocess output:
/ws-stx-i64_20200518/prj_stx/tmp/work/corei7-64-poky-linux/stx-metal/1.0.0-r0/recipe-sysroot-native/usr/sbin/useradd
Running groupadd commands...
NOTE: fm-mgr: Performing groupadd with [--root /ws-stx-i64_20200518/prj_stx/tmp/work/corei7-64-poky-linux/stx-metal/1.0.0-r0/recipe-sysroot -r fm]
NOTE: fm-mgr: group fm already exists, not re-creating it
Running useradd commands...
NOTE: fm-mgr: Performing useradd with [--root /ws-stx-i64_20200518/prj_stx/tmp/work/corei7-64-poky-linux/stx-metal/1.0.0-r0/recipe-sysroot -r -g fm -G snmpd -d /var/lib/fm -s /sbin/nologin -c 'fm-mgr' fm]
useradd: group 'snmpd' does not exist
ERROR: fm-mgr: useradd command did not succeed.



How to reproduce:
=====================
I need add user/group "fm" in my recipe fm-mgr and user/group "snmpd" in recipe net-snmp,
and user "fm" needs to be added in group "snmpd", so what I did includes:

1. Add user/group in recipe net-snmp:

inherit useradd

USERADD_PACKAGES = "net-snmp-server-snmpd"
USERADD_PARAM_net-snmp-server-snmpd = "-r -g snmpd -d /usr/share/snmp -s /sbin/nologin -c 'net-snmp' snmpd"
GROUPADD_PARAM_net-snmp-server-snmpd = "-r snmpd"

2. Add user/group in recipe fm-mgr, and add RDEPENDS on "net-snmp-server-snmpd" since "-G snmpd" is used in useradd parameter:

inherit useradd

USERADD_PACKAGES = "fm-mgr"
USERADD_PARAM_fm-mgr = "-r -g fm -G snmpd -d /var/lib/fm -s /sbin/nologin -c 'fm-mgr' fm"
GROUPADD_PARAM_fm-mgr = "-r fm"

RDEPENDS_fm-mgr += "net-snmp-server-snmpd"


This always work fine in do_rootfs since the dependency ensure the postinst script of "net-snmp-server-snmpd" is run before the one of "fm-mgr", but in do_prepare_recipe_sysroot of recipe that depends on fm-mgr(e.g. stx-metal in this case), postinst-useradd-net-snmp and postinst-useradd-fm-mgr do not run in the order that respect the dependency, so it fails when postinst-useradd-fm-mgr is run before postinst-useradd-net-snmp.
Comment 1 Randy MacLeod 2020-05-21 07:37:14 UTC
Are able to reproduce this on the master branch?
There may be similar useradd bugs in bugzilla already such as perhaps #13279 but we don't have time to check during triage.
Comment 2 Jackie Huang 2020-05-21 07:40:40 UTC
(In reply to comment #1)
> Are able to reproduce this on the master branch?
> There may be similar useradd bugs in bugzilla already such as perhaps #13279
> but we don't have time to check during triage.

Not sure since I haven't had time to try on the master branch yet
Comment 3 Randy MacLeod 2020-06-22 17:49:18 UTC
Jackie, Are you able to work on this defect for M2 or early M3?
Comment 4 Jackie Huang 2020-06-22 17:56:59 UTC
(In reply to comment #3)
> Jackie, Are you able to work on this defect for M2 or early M3?

I'm afraid not, so it's better to have someone else to work on this.
Comment 5 Martin Jansa 2020-10-10 06:49:52 UTC
I have seen similar (if not the same) issue in dunfell based build and added test recipe for it here:

https://patchwork.openembedded.org/series/26477/
https://lists.openembedded.org/g/openembedded-core/message/143163

But here it happens very rarely and I wasn't able to reliably reproduce it.
Comment 6 Ernst Persson 2020-11-02 03:52:06 UTC
It seems to happen once every month or so for me too.
Comment 7 Ernst Persson 2021-05-25 12:55:39 UTC
This is the only random error I'm seeing regularly (but not that often) in Yocto right now.

Is it related to having useradd-staticids ?
Comment 8 Randy MacLeod 2022-03-24 16:02:56 UTC
Is anyone still seeing this bug? Sakib have you tested this recently?

Is this more likely to happen with or without sstate-cache ?
Comment 9 Ernst Persson 2022-03-25 08:56:21 UTC
This is happening to me every other week or so on Dunfell still. I don't have a good way to reproduce it unfortunately and not building on master.
I think you need sstate-cache to reproduce this.
Comment 10 Ernst Persson 2022-05-23 06:46:11 UTC
I've now worked around this by adding extra GROUPADD on depended on groups in the dependant recipe. I can do this since I have staticids. The workaround finally gets rid of the intermittent errors and seems very reliable.
Comment 11 Piotr Łobacz 2023-03-29 15:22:25 UTC
Hi all I have exactly the same issue which I have described here https://bugzilla.yoctoproject.org/show_bug.cgi?id=15084 and yes I'm using sstate_cache
Comment 12 Piotr Łobacz 2023-03-30 07:44:24 UTC
(In reply to Ernst Sjöstrand from comment #10)
> I've now worked around this by adding extra GROUPADD on depended on groups
> in the dependant recipe. I can do this since I have staticids. The
> workaround finally gets rid of the intermittent errors and seems very
> reliable.

I have workaround this exactly the same and it works but it is ugly :P BTW. Thx Ernst
Comment 13 Piotr Łobacz 2023-03-30 19:10:35 UTC
I think I have found the root cause of the issue. Namely there is this base-passwd recipe and in inside it there is this bash script creation:

basepasswd_sysroot_postinst() {
#!/bin/sh

# Install passwd.master and group.master to sysconfdir
install -d -m 755 ${STAGING_DIR_TARGET}${sysconfdir}
for i in passwd group; do
        install -p -m 644 ${STAGING_DIR_TARGET}${datadir}/base-passwd/\$i.master \
                ${STAGING_DIR_TARGET}${sysconfdir}/\$i
done

# Run any useradd postinsts
for script in ${STAGING_DIR_TARGET}${bindir}/postinst-useradd-*; do
        echo \$script
        if [ -f \$script ]; then
                \$script
        fi
done
}

which is running all postinst-useradd-* scripts. Now I haven't verified it yet but I have suspicions that these scripts are launched without an order. Why do I think this way? Because when I had the error that user is missing, a quick workaround for this was just to add missing groups and user to recipe-sysroot/etc/passwd and recipe-sysroot/etc/group files. Another thing is that the same code for user and group add was working for me from recipe which was in alphabetic order and in non-alphabetic it was not... This is still something which I need to confirm.
Comment 14 Piotr Łobacz 2023-03-31 08:58:07 UTC
OK I have made a verification with outputting the status of all the post-inst scripts which are being run and it has turned out that the order of calling them is needed, otherwise an ERROR is occurring e.g.:

/work/build/tmp-glibc/work/core2-64-WelotecGmbH-linux/aziotctl/1.4.3.AUTOINC+cfa39b58ec-r0/recipe-sysroot-native/usr/sbin/useradd
Running groupadd commands...
NOTE: iotedge: Performing groupadd with [--root /work/build/tmp-glibc/work/core2-64-WelotecGmbH-linux/aziotctl/1.4.3.AUTOINC+cfa39b58ec-r0/recipe-sysroot --gid 15580 --system iotedge]
Running useradd commands...
NOTE: iotedge: Performing useradd with [--root /work/build/tmp-glibc/work/core2-64-WelotecGmbH-linux/aziotctl/1.4.3.AUTOINC+cfa39b58ec-r0/recipe-sysroot --comment 'iotedge user' --home-dir /var/lib/aziot/edged --gid 15580 --groups docker,systemd-journal,aziotcs,aziotks,aziottpm,aziotid --system --shell /sbin/nologin --uid 15580 iotedge]
ERROR: iotedge: useradd command did not succeed.
Comment 15 Frédéric Martinsons 2023-04-26 15:22:58 UTC
Hello, 

I came across the exact same issue on dunfell (oe-core: 9aefb4e46cf4fbf14b46f9adaf3771854553e7f3)

It is just to add another acknowledgement to the issue (I have no fix to suggest) and mention that the workaround "adding extra group in GROUPADD" do the job (at least it seems so, since the problem is pretty random)
Comment 16 Louis Rannou 2023-06-23 13:50:10 UTC
Hello,

If you also add :
  DEPENDS += "net-snmp"
or
  DEPENDS:append = " net-snmp "
the issue is not raised.
Comment 17 Martin Jansa 2023-10-05 14:47:21 UTC
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13419 seems to be related
Comment 18 Louis Rannou 2023-10-05 15:14:05 UTC
In my opinion, there are three options:

1. the recipe should specify the dependency (with DEPENDS) to the recipe supposed to create the group
2. useradd tasks should depend on runtime dependencies
3. the recipe should add all required groups to GROUPADD_PARAM

1 and 3 are only documentation but 3 might produce some warning.
Comment 19 Piotr Łobacz 2023-10-05 16:15:35 UTC
Actually, we already have a solution to this problem https://bugzilla.yoctoproject.org/show_bug.cgi?id=15084 but Jan needs to make some slight modifications for this patch which he has not done yet...
Comment 20 Piotr Łobacz 2023-10-05 16:16:20 UTC
(In reply to Piotr Łobacz from comment #19)
> Actually, we already have a solution to this problem
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15084 but Jan needs to
> make some slight modifications for this patch which he has not done yet...

But please you can test it and check if it works for you as well.
Comment 22 Ernst Persson 2024-03-12 13:56:19 UTC
Easy to backport?
Comment 23 Steve Sakoman 2024-03-12 14:01:23 UTC
No idea whether it would be easy or not for any of the stable releases, but please give it a try and submit to the list if you are successful!