11960 2017-08-21 13:07:23 +0000 groupadd fails on ubuntu 16.04 2017-11-07 11:25:53 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.3.1 x86 x86_64 RESOLVED FIXED Medium+ normal 2.5 M1 1 ariede maxin.john bluelightning meta.mr.watcher meta.watcher mikko.ylinen randy.macleod ross.burton --- --- Don't know oldest_to_newest 76027 0 ariede 2017-08-21 13:07:23 +0000 Building core-image-sato on ubuntu 16.04 per quick start instructions. https://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html All packages which inherit a dependency on shadow-native via useradd.bbclass fail groupadd. The groupadd command is not present in the packages' recipe-sysroot-native so the host version is being used and is somehow incompatible. Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for the failed package causes the correct groupadd to be added to recipe-sysroot-native Build Configuration: BB_VERSION = "1.34.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "i586-poky-linux" MACHINE = "qemux86" DISTRO = "poky" DISTRO_VERSION = "2.3.1" TUNE_FEATURES = "m32 i586" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "pyro:4a39979c8d1e560fa54240e99734a651dfbaa63a" Initialising tasks: 100% |##############################################################################################################################################################################| Time: 0:00:11 NOTE: Executing SetScene Tasks NOTE: Executing RunQueue Tasks ERROR: glib-2.0-1_2.50.3-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/home/riedert/poky/meta/classes/staging.bbclass', lineno: 603, function: extend_recipe_sysroot 0599: else: 0600: staging_processfixme(fixme[f], multilibs[f].getVar("RECIPE_SYSROOT"), recipesysroot, recipesysrootnative, d) 0601: 0602: for p in postinsts: *** 0603: subprocess.check_output(p, shell=True) 0604: 0605: for dep in manifests: 0606: c = setscenedeps[dep][0] 0607: os.symlink(manifests[dep], depdir + "/" + c + ".complete") File: '/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output 0622: # empty string. That is maintained here for backwards compatibility. 0623: kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b'' 0624: 0625: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, *** 0626: **kwargs).stdout 0627: 0628: 0629:class CompletedProcess(object): 0630: """A process that has finished running. File: '/usr/lib/python3.5/subprocess.py', lineno: 708, function: run 0704: raise 0705: retcode = process.poll() 0706: if check and retcode: 0707: raise CalledProcessError(retcode, process.args, *** 0708: output=stdout, stderr=stderr) 0709: return CompletedProcess(process.args, retcode, stdout, stderr) 0710: 0711: 0712:def list2cmdline(seq): Exception: subprocess.CalledProcessError: Command '/home/user/poky/build/tmp/work/i586-poky-linux/glib-2.0/1_2.50.3-r0/recipe-sysroot/usr/bin/postinst-useradd-dbus' returned non-zero exit status 1 Subprocess output: /usr/sbin/useradd Running groupadd commands... NOTE: dbus: Performing groupadd with [--root /home/user/poky/build/tmp/work/i586-poky-linux/glib-2.0/1_2.50.3-r0/recipe-sysroot -r netdev] ERROR: dbus: groupadd command did not succeed. 76683 1 ariede 2017-09-08 12:25:11 +0000 It looks like the dependency is being filtered out in sstate.bbclass setscene_depvalid(). # This is due to the [depends] in useradd.bbclass complicating matters # The logic *is* reversed here due to the way hard setscene dependencies are injected if (taskdependees[task][1] == 'do_package' or taskdependees[task][1] == 'do_populate_sysroot') and taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot', 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] == 'do_populate_sysroot': continue https://github.com/openembedded/openembedded-core/blob/2f95c71ba17a6d68d3805efe82097c10b6740244/meta/classes/sstate.bbclass#L949 76760 2 ariede 2017-09-12 13:28:45 +0000 The actual filter that is causing the problem is entitled "# Native/cross tools depended upon by target sysroot are not needed" https://github.com/openembedded/openembedded-core/blob/2f95c71ba17a6d68d3805efe82097c10b6740244/meta/classes/sstate.bbclass#L971 The problem arises from postinst-* scripts which are added to the recipe sysroot from DEPENDS packages (in this case dbus). Said scripts may(do) require native tools which will be filtered out. Removing this filter 'fixed' my problem. Not sure what the wider implications are. 76783 3 maxin.john 2017-09-13 12:06:15 +0000 Managed to reproduce the issue in Ubuntu 16.04 in the first attempt. Strangely, it didn't appear in the second time (bitbake dbus after removing sstate-cache and tmp directories). Currently investigating it. 76791 4 ariede 2017-09-13 15:03:28 +0000 (In reply to comment #3) > Managed to reproduce the issue in Ubuntu 16.04 in the first attempt. > Strangely, it didn't appear in the second time (bitbake dbus after removing > sstate-cache and tmp directories). > > Currently investigating it. In the description it is failing glib-2.0 do_prepare_recipe_sysroot when running postinst-useradd-dbus inherited from the dbus sysroot. 'bitbake glib-2.0' would fail in the same way. 76827 5 maxin.john 2017-09-14 11:37:43 +0000 Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time dependency to glib-2.0. As another check, please add DEPENDS += "shadow-native" to glib-2.0 recipe and see if you could see the same error. 76828 6 ariede 2017-09-14 12:07:16 +0000 (In reply to comment #5) > Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time > dependency to glib-2.0. > > As another check, please add > DEPENDS += "shadow-native" > to glib-2.0 recipe and see if you could see the same error. From initial description: "Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for the failed package causes the correct groupadd to be added to recipe-sysroot-native" 76834 7 maxin.john 2017-09-14 14:42:57 +0000 (In reply to comment #6) > (In reply to comment #5) > > Since ptest is enabled in DISTRO_FEATURES, dbus gets included as a run-time > > dependency to glib-2.0. > > > > As another check, please add > > DEPENDS += "shadow-native" > > to glib-2.0 recipe and see if you could see the same error. > > From initial description: > > "Adding shadow-native (provider of groupadd) explicitly to the DEPENDS for > the failed package causes the correct groupadd to be added to > recipe-sysroot-native" My mistake! I meant, it is a known limitation (need to be fixed) https://bugzilla.yoctoproject.org/show_bug.cgi?id=8078 76967 8 mikko.ylinen 2017-09-19 14:21:26 +0000 (In reply to comment #3) > Managed to reproduce the issue in Ubuntu 16.04 in the first attempt. > Strangely, it didn't appear in the second time (bitbake dbus after removing > sstate-cache and tmp directories). > > Currently investigating it. We started seeing this too and it triggers when the scripts are no longer coming from an sstate (the recent useradd.bbclass changes in master trigger the rebuild). I'm seeing it with 'bitbake -c sdk_depends <image>' (when building esdk). (but not with 'bitbake <image>'. 76981 9 4014 maxin.john 2017-09-20 15:21:56 +0000 Created attachment 4014 fix Please check if this patch fixes the error in these cases. 76993 10 ariede 2017-09-20 20:47:54 +0000 (In reply to comment #9) > Created attachment 4014 [details] > fix > > Please check if this patch fixes the error in these cases. Yes, that works for me. 77003 11 mikko.ylinen 2017-09-21 05:17:37 +0000 (In reply to comment #10) > (In reply to comment #9) > > Created attachment 4014 [details] > > fix > > > > Please check if this patch fixes the error in these cases. > > Yes, that works for me. It won't help here. 77009 12 maxin.john 2017-09-21 08:35:24 +0000 (In reply to comment #11) > (In reply to comment #10) > > (In reply to comment #9) > > > Created attachment 4014 [details] > > > fix > > > > > > Please check if this patch fixes the error in these cases. > > > > Yes, that works for me. > > It won't help here. Just attempted to reproduce the issue in Ubuntu 16.04 using the following command: # bitbake -c sdk_depends core-image-minimal It didn't fail. Could you share the steps to follow here ? 77010 13 mikko.ylinen 2017-09-21 09:26:54 +0000 (In reply to comment #12) > (In reply to comment #11) > > (In reply to comment #10) > > > (In reply to comment #9) > > > > Created attachment 4014 [details] > > > > fix > > > > > > > > Please check if this patch fixes the error in these cases. > > > > > > Yes, that works for me. > > > > It won't help here. > > Just attempted to reproduce the issue in Ubuntu 16.04 using the following > command: > # bitbake -c sdk_depends core-image-minimal > > It didn't fail. Could you share the steps to follow here ? sstate plays a role (and it's not distro related btw). everything worked for us until useradd.bbclass changed recently in OE-core. If I revert 0c4259dd3c74d81f20b87417cecb1e636365ce10 (oe-core commit) I get everything from sstate again. With that commit included setscene is not valid and the error triggers. 77014 14 maxin.john 2017-09-21 10:35:19 +0000 Yes, that commit has the potential to break the sstate. Did we do a clean build (run a cleansstate or remove sstate-cache directory) after that change ? 77221 15 mikko.ylinen 2017-09-29 05:44:58 +0000 (In reply to comment #14) > Yes, that commit has the potential to break the sstate. Did we do a clean > build (run a cleansstate or remove sstate-cache directory) after that change > ? sstate *must* (i.e., do_populate_sysroot_setscene must run for components providing postinst-useradd-* scripts) be used to trigger the problem. I ran bisect and it gave me: 0c4259dd3c74d81f20b87417cecb1e636365ce10 is the first bad commit commit 0c4259dd3c74d81f20b87417cecb1e636365ce10 Author: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Date: Fri Sep 15 16:48:59 2017 +0200 useradd.bbclass: Unify writing of script file in useradd_sysroot_sstate() There were two identical code blocks for writing the postinst-useradd-${PN} scripts. This combines them into one. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> 77223 16 mikko.ylinen 2017-09-29 07:07:10 +0000 (In reply to comment #15) > (In reply to comment #14) > > Yes, that commit has the potential to break the sstate. Did we do a clean > > build (run a cleansstate or remove sstate-cache directory) after that change > > ? > > sstate *must* (i.e., do_populate_sysroot_setscene must run for components > providing postinst-useradd-* scripts) be used to trigger the problem. > Note: the sstate must also come from a mirror where certain paths are different. In case of a failure, the scripts run on my local environment have, e.g., PSEUDO_LOCALSTATEDIR pointing to a path on the CI machine. 77231 17 ariede 2017-09-29 14:39:56 +0000 (In reply to comment #16) > (In reply to comment #15) > > (In reply to comment #14) > > > Yes, that commit has the potential to break the sstate. Did we do a clean > > > build (run a cleansstate or remove sstate-cache directory) after that change > > > ? > > > > sstate *must* (i.e., do_populate_sysroot_setscene must run for components > > providing postinst-useradd-* scripts) be used to trigger the problem. > > > > Note: the sstate must also come from a mirror where certain paths are > different. In case of a failure, the scripts run on my local environment > have, e.g., PSEUDO_LOCALSTATEDIR pointing to a path on the CI machine. (In reply to comment #15) > (In reply to comment #14) > > Yes, that commit has the potential to break the sstate. Did we do a clean > > build (run a cleansstate or remove sstate-cache directory) after that change > > ? > > sstate *must* (i.e., do_populate_sysroot_setscene must run for components > providing postinst-useradd-* scripts) be used to trigger the problem. > > I ran bisect and it gave me: > > 0c4259dd3c74d81f20b87417cecb1e636365ce10 is the first bad commit > commit 0c4259dd3c74d81f20b87417cecb1e636365ce10 > Author: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Date: Fri Sep 15 16:48:59 2017 +0200 > > useradd.bbclass: Unify writing of script file in useradd_sysroot_sstate() > > There were two identical code blocks for writing the > postinst-useradd-${PN} scripts. This combines them into one. > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> > Signed-off-by: Ross Burton <ross.burton@intel.com> That patch looks incorrect. It's intent was apparently to remove the duplicate blocks without affection the functionality, but the implementation removes the code from prepare_recipe_sysroot block. This seems like a different issue. Perhaps it needs a seperate bug? 77256 18 bluelightning 2017-10-01 20:26:32 +0000 (In reply to comment #17) > That patch looks incorrect. It's intent was apparently to remove the > duplicate blocks without affection the functionality, but the implementation > removes the code from prepare_recipe_sysroot block. It removes it yes, but then by virtue of scriptfile being set with the added "if scriptfile:" it will create the file in the block underneath which is identical to that which was removed. At the moment I suspect the reason this is appears to be the cause is that it was simply the last time the signatures changed in a way that triggers the issue. The underlying cause is likely elsewhere. 77300 19 ariede 2017-10-03 12:24:53 +0000 (In reply to comment #18) > (In reply to comment #17) > > That patch looks incorrect. It's intent was apparently to remove the > > duplicate blocks without affection the functionality, but the implementation > > removes the code from prepare_recipe_sysroot block. > > It removes it yes, but then by virtue of scriptfile being set with the added > "if scriptfile:" it will create the file in the block underneath which is > identical to that which was removed. At the moment I suspect the reason this > is appears to be the cause is that it was simply the last time the > signatures changed in a way that triggers the issue. The underlying cause is > likely elsewhere. Sorry, I misread the patch. (In reply to comment #18) > (In reply to comment #17) > > That patch looks incorrect. It's intent was apparently to remove the > > duplicate blocks without affection the functionality, but the implementation > > removes the code from prepare_recipe_sysroot block. > > It removes it yes, but then by virtue of scriptfile being set with the added > "if scriptfile:" it will create the file in the block underneath which is > identical to that which was removed. At the moment I suspect the reason this > is appears to be the cause is that it was simply the last time the > signatures changed in a way that triggers the issue. The underlying cause is > likely elsewhere. It looks bb.build.exec_func("useradd_sysroot", d) will be run before useradd_sysroot is created (part of the moved block). https://github.com/openembedded/openembedded-core/blob/master/meta/classes/useradd.bbclass#L148 77538 20 maxin.john 2017-10-12 08:13:48 +0000 Fix for the first part was sent to oe-core: http://lists.openembedded.org/pipermail/openembedded-core/2017-October/143288.html 78017 21 maxin.john 2017-11-07 11:25:53 +0000 Fix for the reported bug is available in repo: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=06178f64eebdd5cdce4bb8b81f79b4d552af6c0a 4014 2017-09-20 15:21:56 +0000 2017-09-20 15:21:56 +0000 fix 0001-sstate.bbclass-provide-an-exception-for-useradd-scen.patch text/plain 1398 maxin.john RnJvbSAzNmM3NWUzNDhjZDViYjNjZmYyNDI3ODM1NTBiOTI1NjIzNDkyMjk3IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiAiTWF4aW4gQi4gSm9obiIgPG1heGluLmpvaG5AaW50ZWwuY29t PgpEYXRlOiBXZWQsIDIwIFNlcCAyMDE3IDE4OjA1OjQ5ICswMzAwClN1YmplY3Q6IFtQQVRDSF0g c3N0YXRlLmJiY2xhc3M6IHByb3ZpZGUgYW4gZXhjZXB0aW9uIGZvciB1c2VyYWRkIHNjZW5hcmlv CgpQYWNrYWdlcywgd2hpY2ggZGVwZW5kIG9uIHVzZXJzL2dyb3VwcyBjcmVhdGVkIGZyb20gb3Ro ZXIgcGFja2FnZXMsCm5lZWRzICJzaGFkb3ctbmF0aXZlIiBhcyBhIGJ1aWxkIHRpbWUgZGVwZW5k ZW5jeS4gU28sIGFkZCBhbiBleGNlcHRpb24KdG8gdGhlICJzaGFkb3ctbmF0aXZlIiBmcm9tIG90 aGVyd2lzZSBkaXNjYXJkZWQgbmF0aXZlL2Nyb3NzIHRvb2xzCmRlcGVuZGVuY3kuCgpGaXhlcyBb WU9DVE8gIzExOTYwXQoKU2lnbmVkLW9mZi1ieTogTWF4aW4gQi4gSm9obiA8bWF4aW4uam9obkBp bnRlbC5jb20+Ci0tLQogbWV0YS9jbGFzc2VzL3NzdGF0ZS5iYmNsYXNzIHwgMyArKy0KIDEgZmls ZSBjaGFuZ2VkLCAyIGluc2VydGlvbnMoKyksIDEgZGVsZXRpb24oLSkKCmRpZmYgLS1naXQgYS9t ZXRhL2NsYXNzZXMvc3N0YXRlLmJiY2xhc3MgYi9tZXRhL2NsYXNzZXMvc3N0YXRlLmJiY2xhc3MK aW5kZXggMmE1NDk5My4uZTMwZmJlMSAxMDA2NDQKLS0tIGEvbWV0YS9jbGFzc2VzL3NzdGF0ZS5i YmNsYXNzCisrKyBiL21ldGEvY2xhc3Nlcy9zc3RhdGUuYmJjbGFzcwpAQCAtOTY3LDcgKzk2Nyw4 IEBAIGRlZiBzZXRzY2VuZV9kZXB2YWxpZCh0YXNrLCB0YXNrZGVwZW5kZWVzLCBub3RuZWVkZWQs IGQsIGxvZz1Ob25lKToKICAgICAgICAgICAgIGlmIGlzTmF0aXZlQ3Jvc3ModGFza2RlcGVuZGVl c1tkZXBdWzBdKToKICAgICAgICAgICAgICAgICByZXR1cm4gRmFsc2UKICAgICAgICAgICAgICMg TmF0aXZlL2Nyb3NzIHRvb2xzIGRlcGVuZGVkIHVwb24gYnkgdGFyZ2V0IHN5c3Jvb3QgYXJlIG5v dCBuZWVkZWQKLSAgICAgICAgICAgIGlmIGlzTmF0aXZlQ3Jvc3ModGFza2RlcGVuZGVlc1t0YXNr XVswXSk6CisgICAgICAgICAgICAjIEFkZCBhbiBleGNlcHRpb24gZm9yIHNoYWRvdy1uYXRpdmUg YXMgcmVxdWlyZWQgYnkgdXNlcmFkZC5iYmNsYXNzCisgICAgICAgICAgICBpZiBpc05hdGl2ZUNy b3NzKHRhc2tkZXBlbmRlZXNbdGFza11bMF0pIGFuZCB0YXNrZGVwZW5kZWVzW3Rhc2tdWzBdICE9 ICdzaGFkb3ctbmF0aXZlJzoKICAgICAgICAgICAgICAgICBjb250aW51ZQogICAgICAgICAgICAg IyBUYXJnZXQgcG9wdWxhdGVfc3lzcm9vdCBuZWVkIHRoZWlyIGRlcGVuZGVuY2llcwogICAgICAg ICAgICAgcmV0dXJuIEZhbHNlCi0tIAoyLjQuMAoK