4739 2013-06-17 16:02:51 +0000 useradd.bbclass: useradd_sysroot not run per-MACHINE 2013-09-07 08:50:11 +0000 1 1 7 Build System, Metadata & Runtime OE-Core configuration unspecified x86 Multiple RESOLVED FIXED Medium normal 1.5 1 fitzsim richard.purdie jessica.zhang --- --- No (bug/feature does not impact docs) oldest_to_newest 33932 0 fitzsim 2013-06-17 16:02:51 +0000 We use Yocto to build multiple MACHINE images that all use the same TUNE_PKGARCH. We have a TUNE_PKGARCH "users-and-groups" package that installs custom users and groups, and MACHINE_ARCH "application" packages that have DEPENDS on "users-and-groups". We build each machine image with a separate bitbake invocation but starting in the same build directory, e.g.: $ MACHINE=machine1 bitbake machine1-image [...] $ MACHINE=machine2 bitbake machine2-image [...] I'm seeing a problem where sysroots/$machine/etc/passwd is properly updated with our custom users (i.e., "users-and-groups" is properly staged) on the first machine build, but not on subsequent machine builds. This causes build problems in the per-MACHINE "application" packages in the later builds. I've tracked the problem down to do_package stamp files not being handled on a per-machine basis for packages that use useradd.bbclass, so only do_populate_sysroot_setscene is run for the later machines, not do_package_setscene. So for these machines, useradd_sysroot is never run, leaving the sysroot /etc/passwd unmodified. 33938 1 1282 fitzsim 2013-06-17 16:20:34 +0000 Created attachment 1282 useradd.bbclass: Run do_package_setscene per-MACHINE Here is an RFC patch that fixes the issue for my builds. I've also tested it against libnss-mdns/avahi in openembedded-core to confirm that staging of the avahi user still works correctly. I don't think our multi-MACHINE configuration is represented in openembedded-core so I don't think I can provide a better upstream test case. 33939 2 richard.purdie 2013-06-17 16:22:13 +0000 I think this is a use case we're not currently supported or tested (recipe A using users and groups setup by recipe B). do_package is certainly not meant to be machine specific, that would make no sense and cause untold other problems. I'm wondering if this is as simple to fix as changing useradd_sysroot() in useradd.bbclass to list both package_setscene and populate_sysroot_setscene in the conditional? 33940 3 fitzsim 2013-06-17 16:38:12 +0000 Yeah, my first workaround (in the "users-and-groups" recipe) was: useradd_sysroot_sstate () { if [ "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ] then useradd_sysroot fi } do_populate_sysroot_setscene[depends] = "${USERADDSETSCENEDEPS}" but that conflicted with a change you had made to check "package_setscene" instead, so I wasn't sure if manipulating do_populate_sysroot_setscene was a good idea. 33941 4 fitzsim 2013-06-17 16:46:35 +0000 I could test a patch like: diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index 1b419bd..fe64b7f 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -129,7 +129,7 @@ useradd_sysroot () { } useradd_sysroot_sstate () { - if [ "${BB_CURRENTTASK}" = "package_setscene" ] + if [ "${BB_CURRENTTASK}" = "package_setscene" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ] then useradd_sysroot fi and resubmit if this is a use-case you think is worth supporting. Otherwise I can maintain it locally; I filed here in case others are seeing this in their configurations. 33942 5 richard.purdie 2013-06-17 16:50:01 +0000 Give it a go and see if it works? 33943 6 fitzsim 2013-06-17 16:59:29 +0000 OK, will do. 33979 7 richard.purdie 2013-06-19 13:49:04 +0000 (In reply to comment #6) > OK, will do. Any update on whether this worked out of interest? 33981 8 fitzsim 2013-06-19 15:03:37 +0000 (In reply to comment #7) > (In reply to comment #6) > > OK, will do. > > Any update on whether this worked out of interest? Yes, sorry for the delay. Our builds take a long time, and I got caught by the Python 2.7 update on oe-core master. I'll be upgrading my RHEL-5.5-based environment to something more modern soon. For these tests I went back to testing bitbake master and openembedded-core master as of 2013-06-13. But I want to test the patch against tip too, so I'll probably end up building Python 2.7 from source. As for preliminary results, the patch in comment #4 worked for our build, but in oe-core it caused a do_configure error in avahi: NOTE: Executing glib-gettextize --force --copy /distro/build/pure-oe-master-sstate/tmp-eglibc/work/i586-oe-linux/avahi/0.6.31-r6.1/temp/run.do_configure.1288: line 143: glib-gettextize: command not found ERROR: Function failed: do_configure (log file is located at /distro/build/pure-oe-master-sstate/tmp-eglibc/work/i586-oe-linux/avahi/0.6.31-r6.1/temp/log.do_configure.1288) Adding the do_populate_sysroot_setscene[depends] line seems to fix the error. I can't explain why though: diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index e50c889..1aafe06 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -129,7 +129,7 @@ useradd_sysroot () { } useradd_sysroot_sstate () { - if [ "${BB_CURRENTTASK}" = "package_setscene" ] + if [ "${BB_CURRENTTASK}" = "package_setscene" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ] then useradd_sysroot fi @@ -151,6 +151,7 @@ USERADDSETSCENEDEPS_virtclass-cross = "" USERADDSETSCENEDEPS_class-native = "" USERADDSETSCENEDEPS_class-nativesdk = "" do_package_setscene[depends] += "${USERADDSETSCENEDEPS}" +do_populate_sysroot_setscene[depends] += "${USERADDSETSCENEDEPS}" # Recipe parse-time sanity checks def update_useradd_after_parse(d): 33985 9 fitzsim 2013-06-19 22:21:09 +0000 Without the extra do_populate_sysroot_setscene[depends] line, avahi's do_populate_sysroot_setscene task fails: [...] Running useradd commands... grep: /distro/build/pure-oe-master-sstate/tmp-eglibc/sysroots/qemux86/etc/passwd: No such file or directory pseudo: pseudo: can't run useradd: Permission denied [...] and so avahi's do_configure task is attempted: 12019:WARNING: Setscene task 61 (/distro/sources/openembedded-core/meta/recipes-connectivity/avahi/avahi_0.6.31.bb, do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead but glib-2.0-native hasn't been staged by the time it's run, so glib-gettextize isn't available. I still have to retest the dependency addition on my build and on oe-core master. 34396 10 1320 fitzsim 2013-07-08 17:48:39 +0000 Created attachment 1320 useradd.bbclass: Run useradd_sysroot during populate_sysroot_setscene I set up a Python 2.7-based environment and tested this new patch by building libnss-mdns normally and then with shared state, checking the state of avahi users in the sysroot, and comparing with logs from an unpatched build. The patch introduces two changes to the build process: 1) the ordering of avahi _setscene tasks is slightly different to accomodate avahi:do_populate_sysroot_setscene's base-passwd/shadow-native/shadow-sysroot task dependencies. 2) the avahi users are staged immediately by do_populate_sysroot_setscene instead of on-demand by do_package_setscene (e.g. when libnss-mdns is rev-bumped). I think these changes should be harmless. I also tested the patch against our internal distro and it enables the desired use case, where one recipe uses users and groups set up by another recipe, even across multiple MACHINE builds. 36356 11 richard.purdie 2013-09-05 22:40:20 +0000 Sorry for the delayed reply. I'm going to test this patch a little locally and post it on the OE-Core mailing list for review. I think it should be reasonable to merge it. 36383 12 fitzsim 2013-09-06 19:41:31 +0000 Thanks Richard, I'll watch for discussion on the mailing list. 36393 13 richard.purdie 2013-09-07 08:50:11 +0000 Patch merged to master, thanks! http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=f24dda3de90696c5dc33170bcdaa83137cafff9d 1282 2013-06-17 16:20:34 +0000 2013-07-08 17:48:39 +0000 useradd.bbclass: Run do_package_setscene per-MACHINE 0001-useradd.bbclass-Run-do_package_setscene-per-MACHINE.patch text/plain 1095 fitzsim RnJvbSAwYTIyZGE3Njg2OGY4ZDc1NDk4MDgwMTY0ZDU3ODI0YzZjNWVjM2Q1IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBUaG9tYXMgRml0enNpbW1vbnMgPGZpdHpzaW1AY2lzY28uY29t PgpEYXRlOiBTdW4sIDE2IEp1biAyMDEzIDAxOjQ4OjI3IC0wNDAwClN1YmplY3Q6IFtQQVRDSF0g dXNlcmFkZC5iYmNsYXNzOiBSdW4gZG9fcGFja2FnZV9zZXRzY2VuZSBwZXItTUFDSElORQoKRm9y IG11bHRpLU1BQ0hJTkUgYnVpbGRzIHVzaW5nIHRoZSBzYW1lIFRVTkVfUEtHQVJDSCBhbmQgYnVp bGQgZGlyZWN0b3J5LApkb19wYWNrYWdlX3NldHNjZW5lIG11c3QgYmUgcnVuIG9uY2UgcGVyIE1B Q0hJTkUgc3lzcm9vdDsgb3RoZXJ3aXNlIC9ldGMvcGFzc3dkCmlzIG9ubHkgc3RhZ2VkIGNvcnJl Y3RseSBmb3IgdGhlIGZpcnN0IG1hY2hpbmUuCgpbWU9DVE8gIzQ3MzldCgpTaWduZWQtb2ZmLWJ5 OiBUaG9tYXMgRml0enNpbW1vbnMgPGZpdHpzaW1AY2lzY28uY29tPgotLS0KIG1ldGEvY2xhc3Nl cy91c2VyYWRkLmJiY2xhc3MgfCAgICAxICsKIDEgZmlsZXMgY2hhbmdlZCwgMSBpbnNlcnRpb25z KCspLCAwIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy91c2VyYWRkLmJi Y2xhc3MgYi9tZXRhL2NsYXNzZXMvdXNlcmFkZC5iYmNsYXNzCmluZGV4IGU1MGM4ODkuLjFiNDE5 YmQgMTAwNjQ0Ci0tLSBhL21ldGEvY2xhc3Nlcy91c2VyYWRkLmJiY2xhc3MKKysrIGIvbWV0YS9j bGFzc2VzL3VzZXJhZGQuYmJjbGFzcwpAQCAtMTUxLDYgKzE1MSw3IEBAIFVTRVJBRERTRVRTQ0VO RURFUFNfdmlydGNsYXNzLWNyb3NzID0gIiIKIFVTRVJBRERTRVRTQ0VORURFUFNfY2xhc3MtbmF0 aXZlID0gIiIKIFVTRVJBRERTRVRTQ0VORURFUFNfY2xhc3MtbmF0aXZlc2RrID0gIiIKIGRvX3Bh Y2thZ2Vfc2V0c2NlbmVbZGVwZW5kc10gKz0gIiR7VVNFUkFERFNFVFNDRU5FREVQU30iCitkb19w YWNrYWdlW3N0YW1wLWV4dHJhLWluZm9dID0gIiR7TUFDSElORX0iCiAKICMgUmVjaXBlIHBhcnNl LXRpbWUgc2FuaXR5IGNoZWNrcwogZGVmIHVwZGF0ZV91c2VyYWRkX2FmdGVyX3BhcnNlKGQpOgot LSAKMS42LjAuNgoK 1320 2013-07-08 17:48:39 +0000 2013-07-08 17:48:39 +0000 useradd.bbclass: Run useradd_sysroot during populate_sysroot_setscene 0001-useradd.bbclass-Run-useradd_sysroot-during-populate.patch text/plain 1496 fitzsim RnJvbSAyNjQzMmRhNmQzNmYwNjg0MDgwOWYyZmU5YmYwNjY5ZTkwYWE1NjUyIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBUaG9tYXMgRml0enNpbW1vbnMgPGZpdHpzaW1AY2lzY28uY29t PgpEYXRlOiBNb24sIDggSnVsIDIwMTMgMTM6MTc6MDIgLTA0MDAKU3ViamVjdDogW1BBVENIXSB1 c2VyYWRkLmJiY2xhc3M6IFJ1biB1c2VyYWRkX3N5c3Jvb3QgZHVyaW5nIHBvcHVsYXRlX3N5c3Jv b3Rfc2V0c2NlbmUKCkVuYWJsZSBhIG5ldyB1c2VyYWRkIHVzZS1jYXNlOiBvbmUgcmVjaXBlIHVz aW5nIHVzZXJzIGFuZCBncm91cHMgc2V0IHVwIGJ5IGEKYW5vdGhlciByZWNpcGUuCgpXaGVuIG11 bHRpcGxlIE1BQ0hJTkUgYnVpbGRzIHVzZSB0aGUgc2FtZSBUVU5FX1BLR0FSQ0ggYW5kIGJ1aWxk IGRpcmVjdG9yeSwKdXNlcmFkZF9zeXNyb290IG11c3QgYmUgcnVuIG9uY2UgcGVyIE1BQ0hJTkUg c3lzcm9vdDsgb3RoZXJ3aXNlIC9ldGMvcGFzc3dkIGlzCm9ubHkgc3RhZ2VkIGNvcnJlY3RseSAo YnkgZG9fcGFja2FnZV9zZXRzY2VuZSkgZm9yIHRoZSBmaXJzdCBtYWNoaW5lLgoKU2lnbmVkLW9m Zi1ieTogVGhvbWFzIEZpdHpzaW1tb25zIDxmaXR6c2ltQGNpc2NvLmNvbT4KLS0tCiBtZXRhL2Ns YXNzZXMvdXNlcmFkZC5iYmNsYXNzIHwgICAgMyArKy0KIDEgZmlsZXMgY2hhbmdlZCwgMiBpbnNl cnRpb25zKCspLCAxIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy91c2Vy YWRkLmJiY2xhc3MgYi9tZXRhL2NsYXNzZXMvdXNlcmFkZC5iYmNsYXNzCmluZGV4IDNmZTAxMWQu LmNjM2RkZTEgMTAwNjQ0Ci0tLSBhL21ldGEvY2xhc3Nlcy91c2VyYWRkLmJiY2xhc3MKKysrIGIv bWV0YS9jbGFzc2VzL3VzZXJhZGQuYmJjbGFzcwpAQCAtMTg3LDcgKzE4Nyw3IEBAIHVzZXJhZGRf c3lzcm9vdCAoKSB7CiB9CiAKIHVzZXJhZGRfc3lzcm9vdF9zc3RhdGUgKCkgewotCWlmIFsgIiR7 QkJfQ1VSUkVOVFRBU0t9IiA9ICJwYWNrYWdlX3NldHNjZW5lIiBdCisJaWYgWyAiJHtCQl9DVVJS RU5UVEFTS30iID0gInBhY2thZ2Vfc2V0c2NlbmUiIC1vICIke0JCX0NVUlJFTlRUQVNLfSIgPSAi cG9wdWxhdGVfc3lzcm9vdF9zZXRzY2VuZSIgXQogCXRoZW4KIAkJdXNlcmFkZF9zeXNyb290CiAJ ZmkKQEAgLTIwOSw2ICsyMDksNyBAQCBVU0VSQUREU0VUU0NFTkVERVBTX3ZpcnRjbGFzcy1jcm9z cyA9ICIiCiBVU0VSQUREU0VUU0NFTkVERVBTX2NsYXNzLW5hdGl2ZSA9ICIiCiBVU0VSQUREU0VU U0NFTkVERVBTX2NsYXNzLW5hdGl2ZXNkayA9ICIiCiBkb19wYWNrYWdlX3NldHNjZW5lW2RlcGVu ZHNdICs9ICIke1VTRVJBRERTRVRTQ0VORURFUFN9IgorZG9fcG9wdWxhdGVfc3lzcm9vdF9zZXRz Y2VuZVtkZXBlbmRzXSArPSAiJHtVU0VSQUREU0VUU0NFTkVERVBTfSIKIAogIyBSZWNpcGUgcGFy c2UtdGltZSBzYW5pdHkgY2hlY2tzCiBkZWYgdXBkYXRlX3VzZXJhZGRfYWZ0ZXJfcGFyc2UoZCk6 Ci0tIAoxLjYuMC42Cgo=