Building core-image-sato on ubuntu 16.04 per quick start instructions. https://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html All packages which inherit a dependency on shadow-native via useradd.bbclass fail groupadd. The groupadd command is not present in the packages' recipe-sysroot-native so the host version is being used and is somehow incompatible. Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for the failed package causes the correct groupadd to be added to recipe-sysroot-native Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3.1" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "pyro:4a39979c8d1e560fa54240e99734a651dfbaa63a" Initialising tasks: 100% |##############################################################################################################################################################################| Time: 0:00:11 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks ERROR: glib-2.0-1_2.50.3-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/home/riedert/poky/meta/classes/staging.bbclass', lineno: 603, function: extend_recipe_sysroot 0599: else: 0600: staging_processfixme(fixme[f], multilibs[f].getVar("RECIPE_SYSROOT"), recipesysroot, recipesysrootnative, d) 0601: 0602: for p in postinsts: *** 0603: subprocess.check_output(p, shell=True) 0604: 0605: for dep in manifests: 0606: c = setscenedeps[dep][0] 0607: os.symlink(manifests[dep], depdir + "/" + c + ".complete") File: '/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output 0622: # empty string. That is maintained here for backwards compatibility. 0623: kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b'' 0624: 0625: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, *** 0626: **kwargs).stdout 0627: 0628: 0629:class CompletedProcess(object): 0630: """A process that has finished running. File: '/usr/lib/python3.5/subprocess.py', lineno: 708, function: run 0704: raise 0705: retcode = process.poll() 0706: if check and retcode: 0707: raise CalledProcessError(retcode, process.args, *** 0708: output=stdout, stderr=stderr) 0709: return CompletedProcess(process.args, retcode, stdout, stderr) 0710: 0711: 0712:def list2cmdline(seq): Exception: subprocess.CalledProcessError: Command '/home/user/poky/build/tmp/work/i586-poky-linux/glib-2.0/1_2.50.3-r0/recipe-sysroot/usr/bin/postinst-useradd-dbus' returned non-zero exit status 1 Subprocess output: /usr/sbin/useradd Running groupadd commands... NOTE: dbus: Performing groupadd with [--root /home/user/poky/build/tmp/work/i586-poky-linux/glib-2.0/1_2.50.3-r0/recipe-sysroot -r netdev] ERROR: dbus: groupadd command did not succeed.
It looks like the dependency is being filtered out in sstate.bbclass setscene_depvalid(). # This is due to the [depends] in useradd.bbclass complicating matters # The logic *is* reversed here due to the way hard setscene dependencies are injected if (taskdependees[task][1] == 'do_package' or taskdependees[task][1] == 'do_populate_sysroot') and taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot', 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] == 'do_populate_sysroot': continue https://github.com/openembedded/openembedded-core/blob/2f95c71ba17a6d68d3805efe82097c10b6740244/meta/classes/sstate.bbclass#L949
The actual filter that is causing the problem is entitled "# Native/cross tools depended upon by target sysroot are not needed" https://github.com/openembedded/openembedded-core/blob/2f95c71ba17a6d68d3805efe82097c10b6740244/meta/classes/sstate.bbclass#L971 The problem arises from postinst-* scripts which are added to the recipe sysroot from DEPENDS packages (in this case dbus). Said scripts may(do) require native tools which will be filtered out. Removing this filter 'fixed' my problem. Not sure what the wider implications are.
Managed to reproduce the issue in Ubuntu 16.04 in the first attempt. Strangely, it didn't appear in the second time (bitbake dbus after removing sstate-cache and tmp directories). Currently investigating it.
(In reply to comment #3) > Managed to reproduce the issue in Ubuntu 16.04 in the first attempt. > Strangely, it didn't appear in the second time (bitbake dbus after removing > sstate-cache and tmp directories). > > Currently investigating it. In the description it is failing glib-2.0 do_prepare_recipe_sysroot when running postinst-useradd-dbus inherited from the dbus sysroot. 'bitbake glib-2.0' would fail in the same way.
Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time dependency to glib-2.0. As another check, please add DEPENDS += "shadow-native" to glib-2.0 recipe and see if you could see the same error.
(In reply to comment #5) > Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time > dependency to glib-2.0. > > As another check, please add > DEPENDS += "shadow-native" > to glib-2.0 recipe and see if you could see the same error. From initial description: "Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for the failed package causes the correct groupadd to be added to recipe-sysroot-native"
(In reply to comment #6) > (In reply to comment #5) > > Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time > > dependency to glib-2.0. > > > > As another check, please add > > DEPENDS += "shadow-native" > > to glib-2.0 recipe and see if you could see the same error. > > From initial description: > > "Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for > the failed package causes the correct groupadd to be added to > recipe-sysroot-native" My mistake! I meant, it is a known limitation (need to be fixed) https://bugzilla.yoctoproject.org/show_bug.cgi?id=8078
(In reply to comment #3) > Managed to reproduce the issue in Ubuntu 16.04 in the first attempt. > Strangely, it didn't appear in the second time (bitbake dbus after removing > sstate-cache and tmp directories). > > Currently investigating it. We started seeing this too and it triggers when the scripts are no longer coming from an sstate (the recent useradd.bbclass changes in master trigger the rebuild). I'm seeing it with 'bitbake -c sdk_depends <image>' (when building esdk). (but not with 'bitbake <image>'.
Created attachment 4014 [details] fix Please check if this patch fixes the error in these cases.
(In reply to comment #9) > Created attachment 4014 [details] > fix > > Please check if this patch fixes the error in these cases. Yes, that works for me.
(In reply to comment #10) > (In reply to comment #9) > > Created attachment 4014 [details] > > fix > > > > Please check if this patch fixes the error in these cases. > > Yes, that works for me. It won't help here.
(In reply to comment #11) > (In reply to comment #10) > > (In reply to comment #9) > > > Created attachment 4014 [details] > > > fix > > > > > > Please check if this patch fixes the error in these cases. > > > > Yes, that works for me. > > It won't help here. Just attempted to reproduce the issue in Ubuntu 16.04 using the following command: # bitbake -c sdk_depends core-image-minimal It didn't fail. Could you share the steps to follow here ?
(In reply to comment #12) > (In reply to comment #11) > > (In reply to comment #10) > > > (In reply to comment #9) > > > > Created attachment 4014 [details] > > > > fix > > > > > > > > Please check if this patch fixes the error in these cases. > > > > > > Yes, that works for me. > > > > It won't help here. > > Just attempted to reproduce the issue in Ubuntu 16.04 using the following > command: > # bitbake -c sdk_depends core-image-minimal > > It didn't fail. Could you share the steps to follow here ? sstate plays a role (and it's not distro related btw). everything worked for us until useradd.bbclass changed recently in OE-core. If I revert 0c4259dd3c74d81f20b87417cecb1e636365ce10 (oe-core commit) I get everything from sstate again. With that commit included setscene is not valid and the error triggers.
Yes, that commit has the potential to break the sstate. Did we do a clean build (run a cleansstate or remove sstate-cache directory) after that change ?
(In reply to comment #14) > Yes, that commit has the potential to break the sstate. Did we do a clean > build (run a cleansstate or remove sstate-cache directory) after that change > ? sstate *must* (i.e., do_populate_sysroot_setscene must run for components providing postinst-useradd-* scripts) be used to trigger the problem. I ran bisect and it gave me: 0c4259dd3c74d81f20b87417cecb1e636365ce10 is the first bad commit commit 0c4259dd3c74d81f20b87417cecb1e636365ce10 Author: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Date: Fri Sep 15 16:48:59 2017 +0200 useradd.bbclass: Unify writing of script file in useradd_sysroot_sstate() There were two identical code blocks for writing the postinst-useradd-${PN} scripts. This combines them into one. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
(In reply to comment #15) > (In reply to comment #14) > > Yes, that commit has the potential to break the sstate. Did we do a clean > > build (run a cleansstate or remove sstate-cache directory) after that change > > ? > > sstate *must* (i.e., do_populate_sysroot_setscene must run for components > providing postinst-useradd-* scripts) be used to trigger the problem. > Note: the sstate must also come from a mirror where certain paths are different. In case of a failure, the scripts run on my local environment have, e.g., PSEUDO_LOCALSTATEDIR pointing to a path on the CI machine.
(In reply to comment #16) > (In reply to comment #15) > > (In reply to comment #14) > > > Yes, that commit has the potential to break the sstate. Did we do a clean > > > build (run a cleansstate or remove sstate-cache directory) after that change > > > ? > > > > sstate *must* (i.e., do_populate_sysroot_setscene must run for components > > providing postinst-useradd-* scripts) be used to trigger the problem. > > > > Note: the sstate must also come from a mirror where certain paths are > different. In case of a failure, the scripts run on my local environment > have, e.g., PSEUDO_LOCALSTATEDIR pointing to a path on the CI machine. (In reply to comment #15) > (In reply to comment #14) > > Yes, that commit has the potential to break the sstate. Did we do a clean > > build (run a cleansstate or remove sstate-cache directory) after that change > > ? > > sstate *must* (i.e., do_populate_sysroot_setscene must run for components > providing postinst-useradd-* scripts) be used to trigger the problem. > > I ran bisect and it gave me: > > 0c4259dd3c74d81f20b87417cecb1e636365ce10 is the first bad commit > commit 0c4259dd3c74d81f20b87417cecb1e636365ce10 > Author: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Date: Fri Sep 15 16:48:59 2017 +0200 > > useradd.bbclass: Unify writing of script file in useradd_sysroot_sstate() > > There were two identical code blocks for writing the > postinst-useradd-${PN} scripts. This combines them into one. > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Signed-off-by: Ross Burton <ross.burton@intel.com> That patch looks incorrect. It's intent was apparently to remove the duplicate blocks without affection the functionality, but the implementation removes the code from prepare_recipe_sysroot block. This seems like a different issue. Perhaps it needs a seperate bug?
(In reply to comment #17) > That patch looks incorrect. It's intent was apparently to remove the > duplicate blocks without affection the functionality, but the implementation > removes the code from prepare_recipe_sysroot block. It removes it yes, but then by virtue of scriptfile being set with the added "if scriptfile:" it will create the file in the block underneath which is identical to that which was removed. At the moment I suspect the reason this is appears to be the cause is that it was simply the last time the signatures changed in a way that triggers the issue. The underlying cause is likely elsewhere.
(In reply to comment #18) > (In reply to comment #17) > > That patch looks incorrect. It's intent was apparently to remove the > > duplicate blocks without affection the functionality, but the implementation > > removes the code from prepare_recipe_sysroot block. > > It removes it yes, but then by virtue of scriptfile being set with the added > "if scriptfile:" it will create the file in the block underneath which is > identical to that which was removed. At the moment I suspect the reason this > is appears to be the cause is that it was simply the last time the > signatures changed in a way that triggers the issue. The underlying cause is > likely elsewhere. Sorry, I misread the patch. (In reply to comment #18) > (In reply to comment #17) > > That patch looks incorrect. It's intent was apparently to remove the > > duplicate blocks without affection the functionality, but the implementation > > removes the code from prepare_recipe_sysroot block. > > It removes it yes, but then by virtue of scriptfile being set with the added > "if scriptfile:" it will create the file in the block underneath which is > identical to that which was removed. At the moment I suspect the reason this > is appears to be the cause is that it was simply the last time the > signatures changed in a way that triggers the issue. The underlying cause is > likely elsewhere. It looks bb.build.exec_func("useradd_sysroot", d) will be run before useradd_sysroot is created (part of the moved block). https://github.com/openembedded/openembedded-core/blob/master/meta/classes/useradd.bbclass#L148
Fix for the first part was sent to oe-core: http://lists.openembedded.org/pipermail/openembedded-core/2017-October/143288.html
Fix for the reported bug is available in repo: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=06178f64eebdd5cdce4bb8b81f79b4d552af6c0a