So the code in coreutils silently ignores chown permission errors if the caller isn't root.  Operation not support sounds like it tried to set perms but physically couldn't - has the filesystem on the new cluster been changed? Is it possible that it ran the build in a mount that didn't have ownership for some reason?

Note that the error says "cp: preserving permissions ... Operation not supported" which according to the cp.c source in coreutils is the chmod() call, not the chown() call.  The chown would have silently failed as the caller wasn't root, so why would chmod() fail?

I'm thinking the machine was having a big of a moment, maybe the filesystem was in trouble or badly mounted?

This happened again:

https://valkyrie.yoctoproject.org/#/builders/23/builds/185/steps/15/logs/stdio

If I ssh into that machine, su to pokybuild, and strace the command:

openat(AT_FDCWD, "/srv/autobuilder/autobuilder.yocto.io/current_sources/git2_git.yoctoproject.org.git-submodule-test.tar.gz", O_RDONLY) = 3
openat(AT_FDCWD, "foo", O_WRONLY|O_TRUNC) = 4

The source is fd 3, the destination is fd 4.

fgetxattr(3, "system.nfs4_acl", NULL, 0) = 80
fgetxattr(3, "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\36\1\237\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80
fsetxattr(4, "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\36\1\237\0\0\0\6OWNER@\0\0\0\0\0", 80, 0) = -1 EOPNOTSUPP (Operation not supported)

My understanding here is that cp reads the xattrs and then tries to write them, but it can't write system.nfs4_acl to a non-NFS mount (the target is ext4).

So my patch is in fact likely correct, by excluding the xattrs from the preservation.

The outstanding question is why some files have special ACLs set.

Nope, my patch doesn't work - mode implies xattrs if they're set for permissions. But I think we will need to preserve mode for copies of local executables to work.

Bonus fun: fedora39-vk-1 doesn't try to copy the xattrs:

flistxattr(3, NULL, 0)                  = 16
flistxattr(3, "system.nfs4_acl\0", 16)  = 16
openat(AT_FDCWD, "/etc/xattr.conf", O_RDONLY) = 5
newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=817, ...}, AT_EMPTY_PATH) = 0
read(5, "# /etc/xattr.conf\n#\n# Format:\n# "..., 4096) = 817
read(5, "", 4096)                       = 0
close(5)                                = 0
close(4)                                = 0
close(3)                                = 0

Minimal reproducer:

$ cp --preserve=mode /srv/autobuilder/autobuilder.yocto.io/current_sources/git2_git.yoctoproject.org.git-submodule-test.tar.gz foo

This breaks on ubuntu2404-vk but nowhere else.  Downloading packages of coreutils and unpacking them manually, coreutils 9.1 works but 9.4 fails.

Also ubuntu2304-ty works (coreutils 9.1), unless I grab the coreutils 9.4 binary and that fails.

This rules out it being a problem with valkyrie, and identifies it at least being a problem with the coreutils 9.4 in debian/ubuntu.

Fedora 40 also has coreutils 9.4 and this works.

Potentially relevant is https://src.fedoraproject.org/rpms/attr/blob/rawhide/f/0003-attr-2.4.48-xattr-conf-nfs4-acls.patch but the comments suggest the opposite of what we're seeing.

I'd try making the same change to the ubuntu2404 xattr.conf but I've managed to forget my password so I can't sudo to root...

Sent a patch to bitbake: https://lore.kernel.org/bitbake-devel/20240925111046.1286362-1-ross.burton@arm.com/T/#u.

Copy-pasting the commit for anyone reading this:

When copying files as part of the unpack we currently use cp -p, which
is a shortcut for --preserve=mode,ownership,timestamps.

We do want to preserve timestamps, because some fetchers set these
explicitly.

We don't care about ownership. If the files are owned by us then they
ill remain owned by us, and if they're not then the attempt to change
ownership will be silently ignored.  In a shared DL_DIR where files have
group ownership this group access isn't relevant in the single-user build
tree.

We do want to preserve executable bits in the mode, but cp always does
this.  The difference between --preserve=mode and no --preserve is that
the mode isn't preserved exactly (no sticky bits, no suid, umask is
applied) but this also isn't a relevant difference in a build tree.

Also expand the arguments to be clearer about what options are being
passed.

The impetus for this is that coreutils 9.4 includes a change in gnulib[1]
and will now try to preserve permission-based xattrs if asked to preserve
the mode.  This can result in cp failing when copying a file from a
NFSv4 server with ACLs stored in xattrs to a non-NFS directory where
those xattrs cannot be written:

cp: preserving permissions for ‘./jquery-3.7.1.js’: Operation not supported

The error comes from the kernel refusing to write a system.nfs4_acl
xattr to a file on ext4.

This situation doesn't appear on all systems with coreutils 9.4, at the
time of writing it fails on Ubuntu 24.04 onwards but not Fedora 40. This
is because /etc/xattr.conf is used to determine which xattrs describe
permissions, and Fedora 40 has removed the NFSv4 attributes[2].

Merged in 2f35dac0c821ab231459922ed98e1b2cc599ca9a.