1711 2011-11-02 03:55:06 +0000 useradd.bbclass is using UID/GID from sysroots which sometimes doesn't match UID/GID on target 2011-12-16 08:07:39 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core unspecified All Multiple RESOLVED FIXED Patch our for review on mailing list High critical 1.2 1 Martin.Jansa richard.purdie meta.mr.watcher meta.watcher richard.purdie --- --- --- oldest_to_newest 17043 0 Martin.Jansa 2011-11-02 03:55:06 +0000 Updated list of available packages in /var/lib/opkg/lists/jama-om_gta02. Upgrading dbus-1 on root from 1.4.12-r2 to 1.4.12-r7... Downloading http://jama.dyndns-home.com/org.openembedded.shr-core//armv4t/dbus-1_1.4.12-r7_armv4t.ipk. Running groupadd commands... Note: group netdev already exists, not re-creating it Running useradd commands... Note: username messagebus already exists, not re-creating it Upgrading libdbus-1-3 on root from 1.4.12-r2 to 1.4.12-r7... Downloading http://jama.dyndns-home.com/org.openembedded.shr-core//armv4t/libdbus-1-3_1.4.12-r7_armv4t.ipk. Upgrading libglib-2.0-0 on root from 1:2.30.0-r2 to 1:2.30.0-r3... Downloading http://jama.dyndns-home.com/org.openembedded.shr-core//armv4t/libglib-2.0-0_2.30.0-r3_armv4t.ipk. Configuring dbus-1. System startup links for /etc/init.d/dbus-1 already exist. Configuring libdbus-1-3. Configuring libglib-2.0-0. SHR root@gjama / $ ll /usr/libexec/dbus-daemon-launch-helper -rwsr-xr-- 1 root 998 142748 Nov 2 10:12 /usr/libexec/dbus-daemon-launch-helper SHR root@gjama / $ ll -d /var/lib/dbus drwxr-xr-x 2 999 998 4096 Nov 2 10:12 /var/lib/dbus SHR root@gjama / $ grep message /etc/group messagebus:x:101: SHR root@gjama / $ grep message /etc/passwd messagebus:x:42:101:Linux User,,,:/var/run/dbus:/bin/sh and that's because useradd.bbclass is using UID/GID from sysroots OE om-gta02@shr ~/shr-core $ grep message tmp/sysroots/om-gta02/etc/group messagebus:x:998: OE om-gta02@shr ~/shr-core $ grep message tmp/sysroots/om-gta02/etc/passwd messagebus:!:999:998::/var/lib/dbus: and nothing is updating /etc/passwd /etc/group IDs on target. The possible solution would be to force specified UID/GID in useradd.bbclass to make it consistent during rebuild from scratch or teach useradd postinst to update UID/GIDs and chown all runtime created files to new values instead of skiping useradd/groupadd commnads when user already exists, like it did now: Note: username messagebus already exists, not re-creating it Note: group netdev already exists, not re-creating it 17046 1 richard.purdie 2011-11-02 10:50:18 +0000 What should happen is that the names from the tarball in the .ipk should be used in preference to the numeric ids. This isn't happening and a quick look at the opkg source (libbb/unarchive.c) shows that get_tar_header knows about the uid and the uname fields, the uname field is ignored and only the uid field is used. It therefore looks like we need to fix opkg. 17080 2 sgw 2011-11-03 15:51:32 +0000 Problem filed with opkg team. 17191 3 richard.purdie 2011-11-14 04:57:25 +0000 Fixed in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d8193f19fe94224089b0e5fc2026a843f7bd0709 17416 4 Martin.Jansa 2011-11-28 10:08:49 +0000 Just small comment on this implementation: if you're using tar.gz as rootfs you have to be sure to keep UID/GID from .tar.gz (while opkg is trying to keep username/groupnames during upgrades), because you need matching entries in rootfs's /etc/group with files/directories UIDs/GIDs. And tar xzvfp doesn't imply needed --numeric-owner and if you unpack such rootfs ie on card reader in desktop you will probably have different /etc/group and /etc/passwd. 17514 5 Martin.Jansa 2011-12-07 06:36:34 +0000 It's still broken when ie dbus package is used from sstate but base-files are not, (probably because of different checksum) or something like that: that's one explanation why palmpre (also armv7a-vfp-neon like crespo) has root:avahi instead of root:messagebus. ./crespo/shr-lite-20111205-crespo-testlab/files-in-image.txt -rwsr-xr-- 1 root messagebus 184808 Dec 5 01:19 dbus-daemon-launch-helper ./palmpre/shr-lite-20111207-palmpre-testlab/files-in-image.txt -rwsr-xr-- 1 root avahi 184808 Dec 5 01:19 dbus-daemon-launch-helper Other is that sstate extract is missing --numeric-owner too? OE @ ~/shr-core/tmp/sysroots $ diff -uNr crespo/etc/group palmpre/etc/group --- crespo/etc/group 2011-12-06 03:09:18.000000000 +0100 +++ palmpre/etc/group 2011-12-03 01:32:00.000000000 +0100 @@ -37,9 +37,7 @@ games:*:60: users:*:100: nogroup:*:65534: -netdev:x:999: -messagebus:x:998: -avahi:x:997: -sshd:x:996: crontab:x:1000: +avahi:x:998: +sshd:x:997: pulse:x:1001:pulse 17516 6 richard.purdie 2011-12-07 06:56:59 +0000 The point is that everything is consistent within the installed images on the target device as in the correct files are owned by the correct user. The actual numeric IDs themselves aren't so important can can vary so we've never want to force numerical consistency, the names do need to match though. Can you give further details about the problem you're seeing as what you've described so far doesn't sound like a problem... 17517 7 Martin.Jansa 2011-12-07 07:10:14 +0000 (In reply to comment #6) > The point is that everything is consistent within the installed images on the > target device as in the correct files are owned by the correct user. The actual > numeric IDs themselves aren't so important can can vary so we've never want to > force numerical consistency, the names do need to match though. > > Can you give further details about the problem you're seeing as what you've > described so far doesn't sound like a problem... --numeric-owner is important when you're for example preparing uSD card extracting tar.gz image to uSD card on PC or from 2nd partition on target device will by default use owners by name, so the extracted image will be consistent with /etc/group and /etc/passwd on that PC or used on that 2nd partition, but not consistent with just extracted group/passwd (so after reboot ie dbus will fail to autolaunch). And the output from files-in-image.txt shows that sometimes it's not even consistent with coresponding sysroot (I belive that testlab.bbclass is using right sysroot when listing files). 17518 8 richard.purdie 2011-12-07 07:59:22 +0000 Yes, you need to use that when extracting a target image to something like an SD card, I agree its important there. I still don't understand what the problem you're reporting is though :( Yes, different sysroots can have different sets of IDs and that is expected. Are you saying the rootfs set of IDs can be inconsistent with the sysroot set? 17519 9 288 richard.purdie 2011-12-07 08:15:09 +0000 Created attachment 288 Potential fix for image rootfs file ownerhsip mismatch Potential fix for image rootfs file ownerhsip mismatch 17520 10 Martin.Jansa 2011-12-07 09:42:48 +0000 (In reply to comment #9) > Created attachment 288 [details] > Potential fix for image rootfs file ownerhsip mismatch > > Potential fix for image rootfs file ownerhsip mismatch This patch works for me: before: tar -tvf shr-lite-20111207-palmpre.rootfs.tar.gz | grep dbus-daemon-launch -rwsr-xr-- root/avahi 184808 2011-12-05 01:19 //usr/libexec/dbus-daemon-launch-helper after: tar -tvf shr-full-20111207-palmpre.rootfs.tar.gz | grep dbus-daemon-launch -rwsr-xr-- root/messagebus 187568 2011-12-07 17:28 ./usr/libexec/dbus-daemon-launch-helper Same builddir, same repositories just this one patch added 17529 11 richard.purdie 2011-12-08 14:17:51 +0000 Fix merged to master http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2e027278607737aed3c1349eaf6207556ef16bff 17557 12 Martin.Jansa 2011-12-11 05:39:41 +0000 It looks like there are still ways to make owners wrong, some people on ML reported that they still see this issue even after 2e027278607737aed3c1349eaf6207556ef16bff And my builds also confirm that this time it again used wrong group owner (netdev) :/. OE @ ~/shr-core/tmp/deploy/images $ for i in `find . -name files-in-image.txt`; do echo -n $i; grep dbus-daemon-launch $i; done ./om-gta02/shr-full-20111211-om-gta02-testlab/files-in-image.txt-rwsr-xr-- 1 root netdev 142604 Dec 7 13:56 dbus-daemon-launch-helper ./om-gta02/shr-aurora-image-20111211-om-gta02-testlab/files-in-image.txt-rwsr-xr-- 1 root netdev 142604 Dec 7 13:56 dbus-daemon-launch-helper I'll try to find what order of machine builds allows me to reproduce this. 17558 13 Martin.Jansa 2011-12-11 06:35:34 +0000 I was able to reproduce it with oe-core only in core-image-base built from scratch: $ tar -tvf core-image-base-qemux86-64-20111211135551.rootfs.tar.gz | grep dbus-daemon-launch -rwsr-xr-- root/netdev 250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper # user in .ipk is right $ ar x ../x86_64/dbus-1_1.4.16-r0_x86_64.ipk $ tar -tvf data.tar.gz | grep dbus-daemon-launch -rwsr-xr-- root/messagebus 250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper 17559 14 Martin.Jansa 2011-12-11 06:49:03 +0000 More info on this: $ grep "netdev\|messagebus" ../../work/nokia900-oe-linux-gnueabi/shr-image/2.0-r20/rootfs/etc/group netdev:x:998: messagebus:x:997: $ grep "netdev\|messagebus" ../../sysroots/nokia900/etc/group netdev:x:999: messagebus:x:998: 17591 15 Martin.Jansa 2011-12-13 01:59:40 +0000 Current state is that on my 4 armv7a-vfp-neon machines 2 are working only with that patch 2 sometimes working without that patch completely magic.. Today I've tried to cleansstate dbus before running image build and it looks like still using group file from sysroots $ tar -tvf shr-aurora-image-20111213-palmpre.rootfs.tar.gz | grep dbus-daemon-launch -rwsr-xr-- root/995 187568 2011-12-13 09:56 ./usr/libexec/dbus-daemon-launch-helper $ grep 99 ../../../sysroots/palmpre/etc/group avahi:x:998: sshd:x:997: netdev:x:996: messagebus:x:995: xuser:x:994: $ grep 99 ../../../work/palmpre-oe-linux-gnueabi/aurora-image/1.0-r3/rootfs/etc/group xuser:x:999: netdev:x:998: messagebus:x:997: sshd:x:996: 17628 16 Martin.Jansa 2011-12-13 15:16:55 +0000 This is still broken, I'm doing simple test to detect wrong images, but checking owner in tar.gz is not enough as with --numeric-owner it can end again inconsistent with /etc/group for i in `find . -name core\*tar.gz`; do echo $i; tar -tvf $i | grep dbus-daemon-launch; tar --numeric-owner -tvf $i | grep dbus-daemon-launch; tar xzvpf $i ./etc/group; grep messagebus ./etc/group; done | tee -a image.test Here is one image with that last patch and one without, but even with root/messagebus as owner this is not right (see messagebus GID in packaged gropu file) ./core-image-base-qemux86-64-20111211231109.rootfs.tar.gz -rwsr-xr-- root/messagebus 250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper -rwsr-xr-- 0/998 250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper ./etc/group messagebus:x:997: ./core-image-base-qemux86-64-20111211154306.rootfs.tar.gz -rwsr-xr-- root/netdev 250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper -rwsr-xr-- 0/998 250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper ./etc/group messagebus:x:997: So testing for right name is not enough for .tar.gz we need also matching numbers (as --numeric-owner will preserve UID/GID and they don't match /etc/group in image), I'll try to find way to detect wrong jffs2/ubifs images too. 17648 17 richard.purdie 2011-12-14 16:45:18 +0000 Just to dump my status on this into here, I have shared my WIP patch at: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/useradd4&id=7a686fc7c405086bb3182c05c66b3a1da185c8cc The problems in this bug are now constrained to the ipk backend only. The problem is currently opkg does no preinst execution at all the way we use it. We need the preinsts to run before the files are extracted to get the correct permissions. That bit is easy but we also need dependencies to be present when installing packages which is harder as opkg doesn't do this. The patch is a step towards teaching opkg to run the preinst scripts in dependency order. It works but needs cleaning up to work on the target and we should really upgrade opkg at the same time as there are conflicting commits upstream. 17702 18 richard.purdie 2011-12-16 08:07:39 +0000 The opkg preinst issues should be addressed with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1855e94280bdbce2de84cb81cdd61f01950d05d9 288 2011-12-07 08:15:09 +0000 2011-12-07 08:15:09 +0000 Potential fix for image rootfs file ownerhsip mismatch 1 text/plain 1564 richard.purdie ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9pbWFnZS5iYmNsYXNzIGIvbWV0YS9jbGFzc2VzL2lt YWdlLmJiY2xhc3MKaW5kZXggNDY0MmZhNi4uODY1ZDQzMCAxMDA2NDQKLS0tIGEvbWV0YS9jbGFz c2VzL2ltYWdlLmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL2ltYWdlLmJiY2xhc3MKQEAgLTEy MSw2ICsxMjEsOCBAQCBJTUFHRV9MSU5HVUFTID89ICJkZS1kZSBmci1mciBlbi1nYiIKIAogTElO R1VBU19JTlNUQUxMID0gIiR7QCIgIi5qb2luKG1hcChsYW1iZGEgczogImxvY2FsZS1iYXNlLSVz IiAlIHMsIGQuZ2V0VmFyKCdJTUFHRV9MSU5HVUFTJywgMSkuc3BsaXQoKSkpfSIKIAorUFNFVURP X1BBU1NXRCA9ICIke0lNQUdFX1JPT1RGU30iCisKIGRvX3Jvb3Rmc1tub3N0YW1wXSA9ICIxIgog ZG9fcm9vdGZzW2RpcnNdID0gIiR7VE9QRElSfSIKIGRvX3Jvb3Rmc1tsb2NrZmlsZXNdICs9ICIk e0lNQUdFX1JPT1RGU30ubG9jayIKZGlmZiAtLWdpdCBhL21ldGEvY29uZi9iaXRiYWtlLmNvbmYg Yi9tZXRhL2NvbmYvYml0YmFrZS5jb25mCmluZGV4IGU4MGNjMzIuLmVlYjFmYzQgMTAwNjQ0Ci0t LSBhL21ldGEvY29uZi9iaXRiYWtlLmNvbmYKKysrIGIvbWV0YS9jb25mL2JpdGJha2UuY29uZgpA QCAtNTgwLDExICs1ODAsMTIgQEAgU1JDX1VSSSA9ICJmaWxlOi8vJHtGSUxFfSIKIAogIyBVc2Ug cHNldWRvIGFzIHRoZSBmYWtlcm9vdCBpbXBsZW1lbnRhdGlvbgogUFNFVURPX0xPQ0FMU1RBVEVE SVIgPz0gIiR7V09SS0RJUn0vcHNldWRvLyIKK1BTRVVET19QQVNTV0QgPz0gIiR7U1RBR0lOR19E SVJfVEFSR0VUfSIKIGV4cG9ydCBQU0VVRE9fRElTQUJMRUQgPSAiMSIKICNleHBvcnQgUFNFVURP X1BSRUZJWCA9ICIke1NUQUdJTkdfRElSX05BVElWRX0ke3ByZWZpeF9uYXRpdmV9IgogI2V4cG9y dCBQU0VVRE9fQklORElSID0gIiR7U1RBR0lOR19ESVJfTkFUSVZFfSR7YmluZGlyX25hdGl2ZX0i CiAjZXhwb3J0IFBTRVVET19MSUJESVIgPSAiJHtTVEFHSU5HX0RJUl9OQVRJVkV9JFBTRVVET0JJ TkRJUi8uLi9saWIvcHNldWRvL2xpYgotRkFLRVJPT1RFTlYgPSAiUFNFVURPX1BSRUZJWD0ke1NU QUdJTkdfRElSX05BVElWRX0ke3ByZWZpeF9uYXRpdmV9IFBTRVVET19MT0NBTFNUQVRFRElSPSR7 UFNFVURPX0xPQ0FMU1RBVEVESVJ9IFBTRVVET19QQVNTV0Q9JHtTVEFHSU5HX0RJUl9UQVJHRVR9 IFBTRVVET19OT1NZTUxJTktFWFA9MSBQU0VVRE9fRElTQUJMRUQ9MCIKK0ZBS0VST09URU5WID0g IlBTRVVET19QUkVGSVg9JHtTVEFHSU5HX0RJUl9OQVRJVkV9JHtwcmVmaXhfbmF0aXZlfSBQU0VV RE9fTE9DQUxTVEFURURJUj0ke1BTRVVET19MT0NBTFNUQVRFRElSfSBQU0VVRE9fUEFTU1dEPSR7 UFNFVURPX1BBU1NXRH0gUFNFVURPX05PU1lNTElOS0VYUD0xIFBTRVVET19ESVNBQkxFRD0wIgog RkFLRVJPT1ROT0VOViA9ICJQU0VVRE9fVU5MT0FEPTEiCiBGQUtFUk9PVERJUlMgPSAiJHtQU0VV RE9fTE9DQUxTVEFURURJUn0iCiBQUkVGRVJSRURfUFJPVklERVJfdmlydHVhbC9mYWtlcm9vdC1u YXRpdmUgPz0gInBzZXVkby1uYXRpdmUiCg==