Bug 1711

What should happen is that the names from the tarball in the .ipk should be used in preference to the numeric ids. This isn't happening and a quick look at the opkg source (libbb/unarchive.c) shows that get_tar_header knows about the uid and the uname fields, the uname field is ignored and only the uid field is used.

It therefore looks like we need to fix opkg.

Problem filed with opkg team.

Fixed in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d8193f19fe94224089b0e5fc2026a843f7bd0709

Just small comment on this implementation:

if you're using tar.gz as rootfs you have to be sure to keep UID/GID from .tar.gz (while opkg is trying to keep username/groupnames during upgrades), because you need matching entries in rootfs's /etc/group with files/directories UIDs/GIDs.

And tar xzvfp doesn't imply needed --numeric-owner and if you unpack such rootfs ie on card reader in desktop you will probably have different /etc/group and /etc/passwd.

It's still broken when ie dbus package is used from sstate but base-files are not, (probably because of different checksum) or something like that:

that's one explanation why palmpre (also armv7a-vfp-neon like crespo) has root:avahi instead of root:messagebus.

./crespo/shr-lite-20111205-crespo-testlab/files-in-image.txt
-rwsr-xr--  1 root messagebus 184808 Dec  5 01:19 dbus-daemon-launch-helper
./palmpre/shr-lite-20111207-palmpre-testlab/files-in-image.txt
-rwsr-xr--  1 root avahi 184808 Dec  5 01:19 dbus-daemon-launch-helper

Other is that sstate extract is missing --numeric-owner too?

OE @ ~/shr-core/tmp/sysroots $ diff -uNr crespo/etc/group palmpre/etc/group
--- crespo/etc/group    2011-12-06 03:09:18.000000000 +0100
+++ palmpre/etc/group   2011-12-03 01:32:00.000000000 +0100
@@ -37,9 +37,7 @@
 games:*:60:
 users:*:100:
 nogroup:*:65534:
-netdev:x:999:
-messagebus:x:998:
-avahi:x:997:
-sshd:x:996:
 crontab:x:1000:
+avahi:x:998:
+sshd:x:997:
 pulse:x:1001:pulse

The point is that everything is consistent within the installed images on the target device as in the correct files are owned by the correct user. The actual numeric IDs themselves aren't so important can can vary so we've never want to force numerical consistency, the names do need to match though.

Can you give further details about the problem you're seeing as what you've described so far doesn't sound like a problem...

(In reply to comment #6)
> The point is that everything is consistent within the installed images on the
> target device as in the correct files are owned by the correct user. The actual
> numeric IDs themselves aren't so important can can vary so we've never want to
> force numerical consistency, the names do need to match though.
> 
> Can you give further details about the problem you're seeing as what you've
> described so far doesn't sound like a problem...

--numeric-owner is important when you're for example preparing uSD card

extracting tar.gz image to uSD card on PC or from 2nd partition on target device will by default use owners by name, so the extracted image will be consistent with /etc/group and /etc/passwd on that PC or used on that 2nd partition, but not consistent with just extracted group/passwd (so after reboot ie dbus will fail to autolaunch).

And the output from files-in-image.txt shows that sometimes it's not even consistent with coresponding sysroot (I belive that testlab.bbclass is using right sysroot when listing files).

Yes, you need to use that when extracting a target image to something like an SD card, I agree its important there.

I still don't understand what the problem you're reporting is though :(

Yes, different sysroots can have different sets of IDs and that is expected. Are you saying the rootfs set of IDs can be inconsistent with the sysroot set?

Created attachment 288 [details]
Potential fix for image rootfs file ownerhsip mismatch

Potential fix for image rootfs file ownerhsip mismatch

(In reply to comment #9)
> Created attachment 288 [details]
> Potential fix for image rootfs file ownerhsip mismatch
> 
> Potential fix for image rootfs file ownerhsip mismatch

This patch works for me:

before:
tar -tvf shr-lite-20111207-palmpre.rootfs.tar.gz | grep dbus-daemon-launch
-rwsr-xr-- root/avahi   184808 2011-12-05 01:19 //usr/libexec/dbus-daemon-launch-helper

after:
tar -tvf shr-full-20111207-palmpre.rootfs.tar.gz | grep dbus-daemon-launch
-rwsr-xr-- root/messagebus 187568 2011-12-07 17:28 ./usr/libexec/dbus-daemon-launch-helper

Same builddir, same repositories just this one patch added

Fix merged to master http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2e027278607737aed3c1349eaf6207556ef16bff

It looks like there are still ways to make owners wrong, some people on ML reported that they still see this issue even after
2e027278607737aed3c1349eaf6207556ef16bff

And my builds also confirm that this time it again used wrong group owner (netdev) :/.

OE @ ~/shr-core/tmp/deploy/images $ for i in `find . -name files-in-image.txt`; do echo -n $i; grep dbus-daemon-launch $i; done
./om-gta02/shr-full-20111211-om-gta02-testlab/files-in-image.txt-rwsr-xr--  1 root netdev 142604 Dec  7 13:56 dbus-daemon-launch-helper
./om-gta02/shr-aurora-image-20111211-om-gta02-testlab/files-in-image.txt-rwsr-xr--  1 root netdev 142604 Dec  7 13:56 dbus-daemon-launch-helper

I'll try to find what order of machine builds allows me to reproduce this.

I was able to reproduce it with oe-core only in core-image-base built from scratch:

$ tar -tvf core-image-base-qemux86-64-20111211135551.rootfs.tar.gz | grep dbus-daemon-launch
-rwsr-xr-- root/netdev  250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper

# user in .ipk is right
$ ar x ../x86_64/dbus-1_1.4.16-r0_x86_64.ipk
$ tar -tvf data.tar.gz | grep dbus-daemon-launch
-rwsr-xr-- root/messagebus  250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper

More info on this:

$ grep "netdev\|messagebus" ../../work/nokia900-oe-linux-gnueabi/shr-image/2.0-r20/rootfs/etc/group
netdev:x:998:
messagebus:x:997:
$ grep "netdev\|messagebus" ../../sysroots/nokia900/etc/group
netdev:x:999:
messagebus:x:998:

Current state is that on my 4 armv7a-vfp-neon machines
2 are working only with that patch
2 sometimes working without that patch
completely magic..

Today I've tried to cleansstate dbus before running image build and it looks like still using group file from sysroots

$ tar -tvf shr-aurora-image-20111213-palmpre.rootfs.tar.gz | grep dbus-daemon-launch
-rwsr-xr-- root/995     187568 2011-12-13 09:56 ./usr/libexec/dbus-daemon-launch-helper


$ grep 99 ../../../sysroots/palmpre/etc/group 
avahi:x:998:
sshd:x:997:
netdev:x:996:
messagebus:x:995:
xuser:x:994:

$ grep 99 ../../../work/palmpre-oe-linux-gnueabi/aurora-image/1.0-r3/rootfs/etc/group
xuser:x:999:
netdev:x:998:
messagebus:x:997:
sshd:x:996:

This is still broken, I'm doing simple test to detect wrong images, but checking owner in tar.gz is not enough as with --numeric-owner it can end again inconsistent with /etc/group

for i in `find . -name core\*tar.gz`; do 
  echo $i; 
  tar -tvf $i | grep dbus-daemon-launch; 
  tar --numeric-owner -tvf $i | grep dbus-daemon-launch; 
  tar xzvpf $i ./etc/group; 
  grep messagebus ./etc/group; 
done | tee -a image.test

Here is one image with that last patch and one without, but even with root/messagebus as owner this is not right (see messagebus GID in packaged gropu file)

./core-image-base-qemux86-64-20111211231109.rootfs.tar.gz
-rwsr-xr-- root/messagebus  250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper
-rwsr-xr-- 0/998        250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper
./etc/group
messagebus:x:997:
./core-image-base-qemux86-64-20111211154306.rootfs.tar.gz
-rwsr-xr-- root/netdev  250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper
-rwsr-xr-- 0/998        250416 2011-12-10 00:06 ./usr/libexec/dbus-daemon-launch-helper
./etc/group
messagebus:x:997:

So testing for right name is not enough for .tar.gz we need also matching numbers (as --numeric-owner will preserve UID/GID and they don't match /etc/group in image), I'll try to find way to detect wrong jffs2/ubifs images too.

Just to dump my status on this into here, I have shared my WIP patch at:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/useradd4&id=7a686fc7c405086bb3182c05c66b3a1da185c8cc

The problems in this bug are now constrained to the ipk backend only. The problem is currently opkg does no preinst execution at all the way we use it. We need the preinsts to run before the files are extracted to get the correct permissions. That bit is easy but we also need dependencies to be present when installing packages which is harder as opkg doesn't do this.

The patch is a step towards teaching opkg to run the preinst scripts in dependency order. It works but needs cleaning up to work on the target and we should really upgrade opkg at the same time as there are conflicting commits upstream.

The opkg preinst issues should be addressed with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1855e94280bdbce2de84cb81cdd61f01950d05d9