Huh, that's fascinating. How did the file initially get its extended attributes?

Also, does the underlying file actually have those attributes, or were they created under pseudo? My concern is that the kernel may be doing things specially for security.* values, so if they got created in some way that pseudo didn't intercept, the result could be a file which has special attributes magical to the kernel, which pseudo can't necessarily interact with.

The initial systemd file was installed and its meta data updated as part of image creation, i.e. everything happened under sudo.

security.ima was computed by "evmctl ima_sign" in https://github.com/01org/meta-intel-iot-security/blob/master/meta-integrity/classes/ima-evm-rootfs.bbclass. evmctl uses lsetxattr().

security.SMACK64 was set by a Python script which iterates over all files in the rootfs. Because the the "python-xattr" cannot be assumed to be present, some tricks with "ctypes" are needed to call the setxattr() method provided by pseudo. That code is here: https://github.com/pohly/meta-intel-iot-security/blob/smack-fixes/meta-security-framework/classes/xattr-images.bbclass

However, the initial systemd file and also the simplified foobar look perfectly normal when viewed with "ls" under pseudo. It's only after unpacking the tar archive that it gets weird.

Unpacking the archive with "sudo strace tar" on the host looks normal to me:
mknodat(AT_FDCWD, "testrootfs/foobar", 0755) = 0
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=26258, ...}) = 0
mmap(NULL, 26258, PROT_READ, MAP_SHARED, 4, 0) = 0x7fb8fed6f000
close(4)                                = 0
futex(0x7fb8fe7258f8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
setxattr("testrootfs/foobar", "security.ima", "\x03\x02\x02\xd8-N\xfd\x00\x80(M\xe7\x91\x00=0\x1d\x83\x15\xa6N\xbd\xe6\xfe\x13\xa9\x13N\xae\x05B\xb9\x82\xba\x01D\xe6\xf9\xd7\xe2\xf1\x89CsJ\xee3I\x91\xc2W\xbb\x18P\x9cmk\x7f\xaf\x19\xdfY\xa4\x1a\xf9\xdc\xc4\xac\xfd\xdd\x14\xbdF8\xd90\xbe\xd7\xf0\x8d\x07\xe6=\xfeB\xc57c\xc6W\xb3\x18S\x83\xe5\x0e!h\x99WE\xdeV\x02\x88\xbf\x06\x93\x96\xfa\xc4]\xcb\xcd\x8cq\xffJ\xb1Z\xd7\xe3\xc3\x96\xefBv\xbe\x8b\xde\xe3\xb3\x92Ec\xa0K", 137, 0) = 0
setxattr("testrootfs/foobar", "security.SMACK64", "_", 1, 0) = 0
openat(AT_FDCWD, "testrootfs/foobar", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0700) = 4
write(4, "\n", 1)                       = 1
fstat(4, {st_mode=S_IFREG|0755, st_size=1, ...}) = 0
utimensat(4, NULL, {{1453363581, 885600320}, {1453289247, 186050717}}, 0) = 0
fchown(4, 0, 0)                         = 0
fchmod(4, 0755)                         = 0
close(4)                                = 0
close(3)                                = 0
newfstatat(AT_FDCWD, "testrootfs", {st_mode=S_IFDIR|0700, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
utimensat(AT_FDCWD, "testrootfs", {{1453363581, 877600281}, {1453289286, 402245179}}, AT_SYMLINK_NOFOLLOW) = 0
fchownat(AT_FDCWD, "testrootfs", 1000, 1000, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(AT_FDCWD, "testrootfs", 0755)  = 0

My hope was that this unpacking step is something that you might be able to reproduce and (if it leads to the same unexpected result) use to debug the problem.

Just want to get clarification.

The image was generated with xattr items in the image creation scripts with 'pseudo'.  This appears to have worked properly.

You then took the tarball and extracted it with 'sudo' (i.e. to get root permissions and extract the tarball to the filesystem with those same permissions) on the host system and that is when the failure occurred?

This sounds a lot like the host system is preventing you from setting specific attributes.  Perhaps the system is running some type of IMA, Smack or SE Linux code that won't permit you to set these attributes?

A simple check is to use -pseudo- with the tar extraction and in that environment see if the xattrs are what is expected -- if so it's most likely some kind of a host system issue.

(In reply to comment #3)
> The image was generated with xattr items in the image creation scripts with
> 'pseudo'.  This appears to have worked properly.
> 
> You then took the tarball and extracted it with 'sudo' (i.e. to get root
> permissions and extract the tarball to the filesystem with those same
> permissions) on the host system and that is when the failure occurred?

No, that works fine.

> A simple check is to use -pseudo- with the tar extraction and in that
> environment see if the xattrs are what is expected

This is exactly what is failing ;-)

The instructions are not entirely clear, but it was implied that after entering devshell in step 2, you continue with the following commands under pseudo.

That should be easy to reproduce.

The real problem for me is that image creation of an entire rootfs also does not work, but that is harder to reproduce. It might be possible with meta-intel-iot-security, poky and a hand-craft local.conf. Let me know if you want me to try and reproduce it like that.

Huh, this is fascinating.

tar 1.23 doesn't have --xattrs-include.

Using actual root, if I extract that file, and then do getfattrs -d on it, I get:

# getfattr -d -m - testrootfs/foobar
# file: testrootfs/foobar
security.SMACK64="_"

I don't see the .ima value. So something magic is happening when setting that value, it seems?

ohhh. Hmm. I think that may be a red herring, but I'm not sure yet. Because I notice that the failure seems to be that the file's ending up with no file type bits set in its database mode. Hmm. I'm not totally sure the security bits are really what's doing this. It could be an interaction of some sort with the file mode settings.

Anyway, I can reproduce that SOMETHING is going weird here.

Yes, --xattrs is a recent GNU tar feature. You can add ASSUME_PROVIDED_remove = "tar-native" to your local.conf and then "bitbake tar-native" to build it.

Okay, so, it turns out this has nothing at all to do with xattrs, inexplicably enough. Or at least, not directly.

When invoked on that archive with --xattrs --xattrs-include=*, GNU tar uses mknod to create the file instead of open with O_CREAT. And it uses mknod with no file type bits. The pseudo implementation of mknod doesn't check for that, while real mknod masks in S_IFREG if there's no file type bits. So the file gets created with no file type bits, which causes strange behaviors.

I've got a patch to fix this, and I've pushed it to master. I don't know whether that's enough to solve things for now, or whether I should be looking at a new release to put in tree.

Confirmed, http://git.yoctoproject.org/cgit/cgit.cgi/pseudo/commit/?id=735ac1b9ff7471b100f6758f9072bfd45e0e3f40 fixes the tar unpack issue.

The original problem (copying the xattrs from that test rootfs into an ext4 partition) however remains, and it is something that I can only reproduce when running under pseudo. I've file bug 8992 for that.

The pseudo commit which fixes this is present in the upgraded version: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7cd835177ae0208859f790c0c638cde908386d5c

Verified on master 36b43b233c841b3c09fc17e04686b4b758780963"
Not reproducible.