Bug 11960

Summary: groupadd fails on ubuntu 16.04
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Tony Riederer <ariede>
Component: coreAssignee: Maxin B. John <maxin.john>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: bluelightning, meta.mr.watcher, meta.watcher, mikko.ylinen, randy.macleod, ross.burton
Version: 2.3.1   
Target Milestone: 2.5 M1   
Hardware: x86   
OS: x86_64   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
fix none

Description Tony Riederer 2017-08-21 13:07:23 UTC 
Building core-image-sato on ubuntu 16.04 per quick start instructions.
https://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html

All packages which inherit a dependency on shadow-native via useradd.bbclass fail groupadd. 

The groupadd command is not present in the packages' recipe-sysroot-native so the host version is being used and is somehow incompatible.

Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for the failed package causes the correct groupadd to be added to recipe-sysroot-native

Build Configuration:
BB_VERSION        = "1.34.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS        = "i586-poky-linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "2.3.1"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              
meta-poky         
meta-yocto-bsp    = "pyro:4a39979c8d1e560fa54240e99734a651dfbaa63a"

Initialising tasks: 100% |##############################################################################################################################################################################| Time: 0:00:11
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: glib-2.0-1_2.50.3-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/home/riedert/poky/meta/classes/staging.bbclass', lineno: 603, function: extend_recipe_sysroot
     0599:        else:
     0600:            staging_processfixme(fixme[f], multilibs[f].getVar("RECIPE_SYSROOT"), recipesysroot, recipesysrootnative, d)
     0601:
     0602:    for p in postinsts:
 *** 0603:        subprocess.check_output(p, shell=True)
     0604:
     0605:    for dep in manifests:
     0606:        c = setscenedeps[dep][0]
     0607:        os.symlink(manifests[dep], depdir + "/" + c + ".complete")
File: '/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output
     0622:        # empty string. That is maintained here for backwards compatibility.
     0623:        kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
     0624:
     0625:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 *** 0626:               **kwargs).stdout
     0627:
     0628:
     0629:class CompletedProcess(object):
     0630:    """A process that has finished running.
File: '/usr/lib/python3.5/subprocess.py', lineno: 708, function: run
     0704:            raise
     0705:        retcode = process.poll()
     0706:        if check and retcode:
     0707:            raise CalledProcessError(retcode, process.args,
 *** 0708:                                     output=stdout, stderr=stderr)
     0709:    return CompletedProcess(process.args, retcode, stdout, stderr)
     0710:
     0711:
     0712:def list2cmdline(seq):
Exception: subprocess.CalledProcessError: Command '/home/user/poky/build/tmp/work/i586-poky-linux/glib-2.0/1_2.50.3-r0/recipe-sysroot/usr/bin/postinst-useradd-dbus' returned non-zero exit status 1

Subprocess output:
/usr/sbin/useradd
Running groupadd commands...
NOTE: dbus: Performing groupadd with [--root /home/user/poky/build/tmp/work/i586-poky-linux/glib-2.0/1_2.50.3-r0/recipe-sysroot -r netdev]
ERROR: dbus: groupadd command did not succeed.
Comment 1 Tony Riederer 2017-09-08 12:25:11 UTC 
It looks like the dependency is being filtered out in sstate.bbclass setscene_depvalid().

# This is due to the [depends] in useradd.bbclass complicating matters
# The logic *is* reversed here due to the way hard setscene dependencies are injected
if (taskdependees[task][1] == 'do_package' or taskdependees[task][1] == 'do_populate_sysroot') and taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot', 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] == 'do_populate_sysroot':
    continue

https://github.com/openembedded/openembedded-core/blob/2f95c71ba17a6d68d3805efe82097c10b6740244/meta/classes/sstate.bbclass#L949
Comment 2 Tony Riederer 2017-09-12 13:28:45 UTC 
The actual filter that is causing the problem is entitled "# Native/cross tools depended upon by target sysroot are not needed"

https://github.com/openembedded/openembedded-core/blob/2f95c71ba17a6d68d3805efe82097c10b6740244/meta/classes/sstate.bbclass#L971

The problem arises from postinst-* scripts which are added to the recipe sysroot from DEPENDS packages (in this case dbus). Said scripts may(do) require native tools which will be filtered out. 

Removing this filter 'fixed' my problem. Not sure what the wider implications are.
Comment 3 Maxin B. John 2017-09-13 12:06:15 UTC 
Managed to  reproduce the issue in Ubuntu 16.04 in the first attempt. Strangely, it didn't appear in the second time (bitbake dbus after removing sstate-cache and tmp directories).

Currently investigating it.
Comment 4 Tony Riederer 2017-09-13 15:03:28 UTC 
(In reply to comment #3)
> Managed to  reproduce the issue in Ubuntu 16.04 in the first attempt.
> Strangely, it didn't appear in the second time (bitbake dbus after removing
> sstate-cache and tmp directories).
> 
> Currently investigating it.

In the description it is failing glib-2.0 do_prepare_recipe_sysroot when running postinst-useradd-dbus inherited from the dbus sysroot. 'bitbake glib-2.0' would fail in the same way.
Comment 5 Maxin B. John 2017-09-14 11:37:43 UTC 
Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time dependency to glib-2.0.

As another check, please add 
DEPENDS += "shadow-native"
to glib-2.0 recipe and see if you could see the same error.
Comment 6 Tony Riederer 2017-09-14 12:07:16 UTC 
(In reply to comment #5)
> Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time
> dependency to glib-2.0.
> 
> As another check, please add 
> DEPENDS += "shadow-native"
> to glib-2.0 recipe and see if you could see the same error.

From initial description: 

"Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for the failed package causes the correct groupadd to be added to recipe-sysroot-native"
Comment 7 Maxin B. John 2017-09-14 14:42:57 UTC 
(In reply to comment #6)
> (In reply to comment #5)
> > Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time
> > dependency to glib-2.0.
> > 
> > As another check, please add 
> > DEPENDS += "shadow-native"
> > to glib-2.0 recipe and see if you could see the same error.
> 
> From initial description: 
> 
> "Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for
> the failed package causes the correct groupadd to be added to
> recipe-sysroot-native"

My mistake!

I meant, it is a known limitation (need to be fixed)
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8078
Comment 8 Mikko Ylinen 2017-09-19 14:21:26 UTC 
(In reply to comment #3)
> Managed to  reproduce the issue in Ubuntu 16.04 in the first attempt.
> Strangely, it didn't appear in the second time (bitbake dbus after removing
> sstate-cache and tmp directories).
> 
> Currently investigating it.


We started seeing this too and it triggers when the scripts are no longer coming from an sstate (the recent useradd.bbclass changes in master trigger the rebuild). I'm seeing it with 'bitbake -c sdk_depends <image>' (when building esdk). (but not with 'bitbake <image>'.
Comment 9 Maxin B. John 2017-09-20 15:21:56 UTC 
Created attachment 4014 [details]
fix

Please check if this patch fixes the error in these cases.
Comment 10 Tony Riederer 2017-09-20 20:47:54 UTC 
(In reply to comment #9)
> Created attachment 4014 [details]
> fix
> 
> Please check if this patch fixes the error in these cases.

Yes, that works for me.
Comment 11 Mikko Ylinen 2017-09-21 05:17:37 UTC 
(In reply to comment #10)
> (In reply to comment #9)
> > Created attachment 4014 [details]
> > fix
> > 
> > Please check if this patch fixes the error in these cases.
> 
> Yes, that works for me.

It won't help here.
Comment 12 Maxin B. John 2017-09-21 08:35:24 UTC 
(In reply to comment #11)
> (In reply to comment #10)
> > (In reply to comment #9)
> > > Created attachment 4014 [details]
> > > fix
> > > 
> > > Please check if this patch fixes the error in these cases.
> > 
> > Yes, that works for me.
> 
> It won't help here.

Just attempted to reproduce the issue in Ubuntu 16.04 using the following command:
# bitbake -c sdk_depends core-image-minimal

It didn't fail. Could you share the steps to follow here ?
Comment 13 Mikko Ylinen 2017-09-21 09:26:54 UTC 
(In reply to comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > (In reply to comment #9)
> > > > Created attachment 4014 [details]
> > > > fix
> > > > 
> > > > Please check if this patch fixes the error in these cases.
> > > 
> > > Yes, that works for me.
> > 
> > It won't help here.
> 
> Just attempted to reproduce the issue in Ubuntu 16.04 using the following
> command:
> # bitbake -c sdk_depends core-image-minimal
> 
> It didn't fail. Could you share the steps to follow here ?

sstate plays a role (and it's not distro related btw). everything worked for us until useradd.bbclass changed recently in OE-core. If I revert 0c4259dd3c74d81f20b87417cecb1e636365ce10 (oe-core commit) I get everything from sstate again. With that commit included setscene is not valid and the error triggers.
Comment 14 Maxin B. John 2017-09-21 10:35:19 UTC 
Yes, that commit has the potential to break the sstate. Did we do a clean build (run a cleansstate or remove sstate-cache directory) after that change ?
Comment 15 Mikko Ylinen 2017-09-29 05:44:58 UTC 
(In reply to comment #14)
> Yes, that commit has the potential to break the sstate. Did we do a clean
> build (run a cleansstate or remove sstate-cache directory) after that change
> ?

sstate *must* (i.e., do_populate_sysroot_setscene must run for components providing postinst-useradd-* scripts) be used to trigger the problem.

I ran bisect and it gave me:

0c4259dd3c74d81f20b87417cecb1e636365ce10 is the first bad commit
commit 0c4259dd3c74d81f20b87417cecb1e636365ce10
Author: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Date:   Fri Sep 15 16:48:59 2017 +0200

    useradd.bbclass: Unify writing of script file in useradd_sysroot_sstate()

    There were two identical code blocks for writing the
    postinst-useradd-${PN} scripts. This combines them into one.

    Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
Comment 16 Mikko Ylinen 2017-09-29 07:07:10 UTC 
(In reply to comment #15)
> (In reply to comment #14)
> > Yes, that commit has the potential to break the sstate. Did we do a clean
> > build (run a cleansstate or remove sstate-cache directory) after that change
> > ?
> 
> sstate *must* (i.e., do_populate_sysroot_setscene must run for components
> providing postinst-useradd-* scripts) be used to trigger the problem.
> 

Note: the sstate must also come from a mirror where certain paths are different. In case of a failure, the scripts run on my local environment have, e.g., PSEUDO_LOCALSTATEDIR pointing to a path on the CI machine.
Comment 17 Tony Riederer 2017-09-29 14:39:56 UTC 
(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #14)
> > > Yes, that commit has the potential to break the sstate. Did we do a clean
> > > build (run a cleansstate or remove sstate-cache directory) after that change
> > > ?
> > 
> > sstate *must* (i.e., do_populate_sysroot_setscene must run for components
> > providing postinst-useradd-* scripts) be used to trigger the problem.
> > 
> 
> Note: the sstate must also come from a mirror where certain paths are
> different. In case of a failure, the scripts run on my local environment
> have, e.g., PSEUDO_LOCALSTATEDIR pointing to a path on the CI machine.

(In reply to comment #15)
> (In reply to comment #14)
> > Yes, that commit has the potential to break the sstate. Did we do a clean
> > build (run a cleansstate or remove sstate-cache directory) after that change
> > ?
> 
> sstate *must* (i.e., do_populate_sysroot_setscene must run for components
> providing postinst-useradd-* scripts) be used to trigger the problem.
> 
> I ran bisect and it gave me:
> 
> 0c4259dd3c74d81f20b87417cecb1e636365ce10 is the first bad commit
> commit 0c4259dd3c74d81f20b87417cecb1e636365ce10
> Author: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Date:   Fri Sep 15 16:48:59 2017 +0200
> 
>     useradd.bbclass: Unify writing of script file in useradd_sysroot_sstate()
> 
>     There were two identical code blocks for writing the
>     postinst-useradd-${PN} scripts. This combines them into one.
> 
>     Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
>     Signed-off-by: Ross Burton <ross.burton@intel.com>

That patch looks incorrect. It's intent was apparently to remove the duplicate blocks without affection the functionality, but the implementation removes the code from prepare_recipe_sysroot block. This seems like a different issue. Perhaps it needs a seperate bug?
Comment 18 Paul Eggleton 2017-10-01 20:26:32 UTC 
(In reply to comment #17)
> That patch looks incorrect. It's intent was apparently to remove the
> duplicate blocks without affection the functionality, but the implementation
> removes the code from prepare_recipe_sysroot block.

It removes it yes, but then by virtue of scriptfile being set with the added "if scriptfile:" it will create the file in the block underneath which is identical to that which was removed. At the moment I suspect the reason this is appears to be the cause is that it was simply the last time the signatures changed in a way that triggers the issue. The underlying cause is likely elsewhere.
Comment 19 Tony Riederer 2017-10-03 12:24:53 UTC 
(In reply to comment #18)
> (In reply to comment #17)
> > That patch looks incorrect. It's intent was apparently to remove the
> > duplicate blocks without affection the functionality, but the implementation
> > removes the code from prepare_recipe_sysroot block.
> 
> It removes it yes, but then by virtue of scriptfile being set with the added
> "if scriptfile:" it will create the file in the block underneath which is
> identical to that which was removed. At the moment I suspect the reason this
> is appears to be the cause is that it was simply the last time the
> signatures changed in a way that triggers the issue. The underlying cause is
> likely elsewhere.

Sorry, I misread the patch. 

(In reply to comment #18)
> (In reply to comment #17)
> > That patch looks incorrect. It's intent was apparently to remove the
> > duplicate blocks without affection the functionality, but the implementation
> > removes the code from prepare_recipe_sysroot block.
> 
> It removes it yes, but then by virtue of scriptfile being set with the added
> "if scriptfile:" it will create the file in the block underneath which is
> identical to that which was removed. At the moment I suspect the reason this
> is appears to be the cause is that it was simply the last time the
> signatures changed in a way that triggers the issue. The underlying cause is
> likely elsewhere.

It looks bb.build.exec_func("useradd_sysroot", d) will be run before useradd_sysroot is created (part of the moved block). 

https://github.com/openembedded/openembedded-core/blob/master/meta/classes/useradd.bbclass#L148
Comment 20 Maxin B. John 2017-10-12 08:13:48 UTC 
Fix for the first part was sent to oe-core:
http://lists.openembedded.org/pipermail/openembedded-core/2017-October/143288.html
Comment 21 Maxin B. John 2017-11-07 11:25:53 UTC 
Fix for the reported bug is available in repo:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=06178f64eebdd5cdce4bb8b81f79b4d552af6c0a