| Summary: | AB-INT: do_image_wic: tar command return exit status 2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Antonin Godard <antonin.godard> | ||||||
| Component: | oe-selftest | Assignee: | Paul Barker <paul> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | High | CC: | mathieu.dubois-briand, paul, randy.macleod, richard.purdie, yoann.congal | ||||||
| Version: | unspecified | ||||||||
| Target Milestone: | 6.0 M3 | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| See Also: |
https://bugzilla.yoctoproject.org/show_bug.cgi?id=16117 https://bugzilla.yoctoproject.org/show_bug.cgi?id=16156 |
||||||||
| Whiteboard: | AB-INT | ||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||
| Attachments: |
|
||||||||
|
Description
Antonin Godard
2026-01-19 15:12:25 UTC
oe-selftest-fedora rocky9-vk-2 agodard/master-next completed at 2026-01-15 14:06:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2913/steps/15/logs/stdio oe-selftest-fedora rocky9-vk-2 agodard/master-next completed at 2026-01-16 15:59:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2930/steps/15/logs/stdio The openat2() change in in pseudo and pseudo has been updated. oe-selftest-fedora alma9-vk-2 master completed at 2026-01-25 02:31:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2991/steps/15/logs/stdio This happened again yesterday on an alma9 worker -> re-open oe-selftest-fedora stream9-vk-1 mathieu/master-next completed at 2026-01-28 06:21:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3001/steps/15/logs/stdio oe-selftest-fedora stream9-vk-1 mathieu/master-next completed at 2026-01-29 23:18:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3010/steps/15/logs/stdio oe-selftest-fedora alma9-vk-2 master completed at 2026-02-03 02:33:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3033/steps/15/logs/stdio oe-selftest-fedora rocky9-vk-2 master completed at 2026-02-10 03:12:05+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3057/steps/15/logs/stdio qemuarm-oecore alma9-vk-2 mathieu/master-next completed at 2026-02-11 21:40:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3142/steps/12/logs/stdio scarthgap qemuarm-oecore rocky9-vk-1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/40/builds/3140 ^ that is a stable/scarthgap-nut branch *with the pseudo 1.9.3 upgrade* Created attachment 5183 [details]
Minimal reproducer
Created attachment 5184 [details]
pseudo.log
We were able to create a minimal reproducer (attached) from the build directory left behind by https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3142/steps/12/logs/stdio. The alma9-vk-2 worker is still paused, with the full build directory in /srv/pokybuild/yocto-worker/qemuarm-oecore/build if the minimal reproducer isn't sufficient for debugging. In the case of failing do_package tasks under oe-selftest, the pseudo output was likely being swallowed but the failure condition is still the same - a tar invocation fails with exit code 2 under pseudo. This is always occuring on RHEL 9 derivatives so we're again dealing with the backported openat2 implementation. For CentOS Stream 9 we can see the offending source at https://gitlab.com/redhat/centos-stream/rpms/tar/-/blob/c9s/tar-1.34-CVE-2025-45582.patch?ref_type=heads, I think it's reasonable to assume the same code is being used in AlmaLinux 9 and Rocky Linux 9. So, the actual issue: We've modified pseudo to intercept openat2 calls and return -1 with errno=ENOSYS. We never pass on the openat2 call to glibc, so the actual openat2 syscall should never occur. However, looking at the attached log file where strace was used, we do see an openat2 syscall: > openat2(AT_FDCWD, "info/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = 4 This results in fd 4 being opened. Subsequent calls then fail as pseudo doesn't know the path corresponding to fd 4 as the above openat2 call was never intercepted. > 257933: got *at() syscall for unknown directory, fd 4 > 257933: unknown base path for fd 4, path pre-applypatch.sample > 257933: couldn't allocate absolute path for 'pre-applypatch.sample'. > 257933: openat_creat: (null) -> 0755 > tar: ./hooks/pre-applypatch.sample: Cannot open: Bad address This causes tar to fail with an exit code 2. We need to figure out how these openat2 calls are bypassing pseudo and see if we can intercept them. One of the tar patches someone sent me, they implemented openat2 directly in tar itself. So tar was calling it's own 'openat2', not the glibc one (since there wasn't one), and then calling the syscall itself. So the only thing you should be able to see is the syscall being invoked. The syscall intercept in pseudo should be able to stop that, assuming they're not playing further games and avoiding glibc in the syscall. The syscall wrapper has the following code:
pseudo_wrappers.c:
#ifdef SYS_openat2
/* There is a CVE patch (CVE-2025-45582) to tar 1.34 in Centos Stream which
* uses syscall to access openat2() and breaks builds if we don't redirect.
*/
if (number == SYS_openat2) {
va_start(ap, number);
int dirfd = va_arg(ap, int);
const char * path = va_arg(ap, const char *);
void *how = va_arg(ap, void *);
size_t size = va_arg(ap, size_t);
return wrap_openat2(dirfd, path, how, size);
}
#endif
My suggestion is to:
* Verify SYS_openat2 is actually defined. It was on the prior test machines. If it's not defined, we may need to define it inline to the right value.
* Add instrumentation to the syscall function so when it's called directly we get notified somehow.
To clarify Mark's point, we need to check whether the openat2 syscall number is defined on all our host distros. Currently theory is that pseudo-native could have been built on a system where that syscall isn't defined. If so, we should add some fall back defaults. I used parallel-ssh to run "grep openat2 /usr/include/asm-generic/unistd.h" on all the autobuilder workers and they all show: #define __NR_openat2 437 __SYSCALL(__NR_openat2, sys_openat2) i.e. it is defined everywhere Sorry, that wasn't what Mark asked for. Running "grep -R SYS_openat2 /usr/include/" on the machines showed three that don't have it, debian11-vk-1, debian11-vk3 and opensuse155-vk-2.yocto.io. The define is in different places in different kernels but looks to be missing on these three. Those have for example: rpurdie@debian11-vk-1:/usr/include$ grep openat2 -R x86_64-linux-gnu/asm/unistd_64.h:#define __NR_openat2 437 x86_64-linux-gnu/asm/unistd_x32.h:#define __NR_openat2 (__X32_SYSCALL_BIT + 437) x86_64-linux-gnu/asm/unistd_32.h:#define __NR_openat2 437 linux/openat2.h: * Arguments for how openat2(2) should open the target path. If only @flags and linux/openat2.h: * @mode are non-zero, then openat2(2) operates very similarly to openat(2). linux/openat2.h:/* how->resolve flags for openat2(2). */ linux/fcntl.h:#include <linux/openat2.h> asm-generic/unistd.h:#define __NR_openat2 437 asm-generic/unistd.h:__SYSCALL(__NR_openat2, sys_openat2) so we probably need a fallback definition of the value. My recommendation is we just say we require a "new enough" kernel that supports openat2, and always define the value in: ports/linux/pseudo_wrappers.c The openat2 call wrapper already is unconditional. (Included by ports/linux/subports) Hi Mark, RHEL 8 and derivatives use Linux 4.18, this has maintenance support for a while yet (https://endoflife.date/rhel). openat2 was introduced in Linux 5.6. I don't think that's an obstacle though as we can return ENOSYS and let the caller handle it. With the recent commit: ports/linux: Ensure SYS_openat2 is defined - pseudo - Allows emulation of root user operations https://git.yoctoproject.org/pseudo/commit/?id=899736ef46bb903099e13d649cfe785c29135cab ... merged on scarthgap, a user reported that it broke the build on Ubuntu 20.04. This is because Ubuntu 20.04 is on Linux 5.4 and does not have a __NR_openat2. I've sent a patch to define it in pseudo: [pseudo] [PATCH] ports/linux: define __NR_openat2 if missing https://lists.yoctoproject.org/g/yocto-patches/message/3248 The fix for Ubuntu 20.04 has merged: https://git.yoctoproject.org/pseudo/commit/?id=43cbd8fb4914328094ccdb4bb827d74b1bac2046 https://git.openembedded.org/openembedded-core/commit/?id=e9a35f32b983de724d2c2e436c017b49d5b70469 So, we can close this :) Argh! :( oe-selftest-fedora alma9-vk-2 mathieu/master-next completed at 2026-03-12 12:26:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3279/steps/15/logs/stdio Reopening, logs incoming... And once again on alma9-vk-2: oe-selftest-fedora alma9-vk-2 master completed at 2026-03-18 02:45:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3336/steps/15/logs/stdio oe-selftest-fedora alma9-vk-2 master completed at 2026-03-17 05:53:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3321/steps/15/logs/stdio And another one on rocky9-vk-2: oe-selftest-fedora rocky9-vk-2 mathieu/master-next completed at 2026-03-18 11:56:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3337/steps/15/logs/stdio oe-selftest-fedora rocky9-vk-2 mathieu/master-next completed at 2026-03-18 13:58:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3342/steps/15/logs/stdio Richard says that this is sometimes happening for him. It's mainly (only ?) on rocky/alma hosts and may not involve pseudo. After more debugging, we have a likely culprit: "extracting using --xattrs of read only files lead to Cannot open: Permission denied issue" https://bugzilla.redhat.com/show_bug.cgi?id=1886540 "Fix bug with -x --xattr read-only files" https://cgit.git.savannah.gnu.org/cgit/tar.git/commit/?id=0b74885e81b90d6ab4890b195dce99ca9109fe59 From Paul, we can reproduce this with tar directly on the stream9/rocky9 systems: [root@stream9 ~]# dnf update Last metadata expiration check: 0:10:04 ago on Fri 20 Mar 2026 07:59:08 AM UTC. Dependencies resolved. Nothing to do. Complete! [root@stream9 ~]# su - testuser Last login: Thu Mar 19 18:08:58 UTC 2026 on pts/1 [testuser@stream9 ~]$ mkdir -p dir1/dir2 [testuser@stream9 ~]$ touch dir1/dir2/file1 [testuser@stream9 ~]$ setfattr -n user.foo -v bar dir1/dir2/file1 [testuser@stream9 ~]$ chmod 0440 dir1/dir2/file1 [testuser@stream9 ~]$ mkdir /tmp/testdir [testuser@stream9 ~]$ tar --xattrs -cf /tmp/testdir/file.tar dir1/dir2/file1 [testuser@stream9 ~]$ cd /tmp/testdir/ [testuser@stream9 testdir]$ tar --xattrs -xf /tmp/testdir/file.tar dir1/dir2/file1 tar: setxattrat: Cannot set 'user.foo' extended attribute for file 'dir1/dir2/file1': Permission denied tar: dir1/dir2/file1: Cannot open: Permission denied tar: Exiting with failure status due to previous errors To be clear there were two issues here, a pseudo problem with the openat2 call which we've fixed and then a secondary issue only in tar in the steam9/rocky9 systems. oe-selftest-fedora stream9-vk-1 mathieu/master-next completed at 2026-03-19 21:59:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3360/steps/15/logs/stdio oe-selftest-fedora stream9-vk-1 mathieu/master-next completed at 2026-03-20 10:42:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3366/steps/15/logs/stdio oe-selftest-fedora alma9-vk-2 mathieu/master-next completed at 2026-03-20 22:08:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3372/steps/15/logs/stdio oe-selftest-fedora stream9-vk-1 master completed at 2026-03-21 03:48:09+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3375/steps/15/logs/stdio a-full fedora43-vk-2 mathieu/master-next completed at 2026-03-28 08:00:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3540/steps/5/logs/stdio a-full alma8-vk-2 mathieu/master-next completed at 2026-03-28 11:22:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3541/steps/5/logs/stdio So, after discussion with CentOS folks, this is now tracked for RHEL 9: https://redhat.atlassian.net/browse/RHEL-158896 A change in RHEL 9 or related distros to fix this is unlikely at this stage. It's not a regresssion, and it's not considered a major issue impacting customers. Any fix risks breaking existing workarounds others may have in place. We're likely stuck with this breakage :( That means the only option is to recommend buildtools on affected systems. I think the next steps are: 1) Enable buildtools on stream9, alma9 & rocky9 workers. 2) Confirm with multiple builds that the issue is gone. 3) Update documentation to tell users of these systems to use buildtools. Patch sent for the autobuilder: https://lore.kernel.org/yocto-patches/20260331-rhel9-v1-1-e5c2cf815386@pbarker.dev/T/#u This will also need backporting to stable branches. I confirmed that tar from buildtools works on rocky9 with a tarball that trips up the native tar: (buildbot-venv) [pokybuild@rocky9-vk-2 x]$ tar --xattrs -xf ../test.tar tar: setxattrat: Cannot set 'user.test' extended attribute for file 'sourcedir/hello.txt': Permission denied tar: sourcedir/hello.txt: Cannot open: Permission denied tar: Exiting with failure status due to previous errors (buildbot-venv) [pokybuild@rocky9-vk-2 x]$ rm -rf sourcedir/ (buildbot-venv) [pokybuild@rocky9-vk-2 x]$ source ../openembedded-core/buildtools/environment-setup-x86_64-pokysdk-linux (buildbot-venv) [pokybuild@rocky9-vk-2 x]$ which tar ~/tar-test/openembedded-core/buildtools/sysroots/x86_64-pokysdk-linux/usr/bin/tar (buildbot-venv) [pokybuild@rocky9-vk-2 x]$ tar --xattrs -xf ../test.tar (buildbot-venv) [pokybuild@rocky9-vk-2 x]$ We do need to add something to sanity.bbclass to detect the problematic situation too Patch sent for the docs: https://lore.kernel.org/r/20260402-alma-centos-rocky-builtools-v1-1-447c92372100@bootlin.com Fixed by enforcing a tar upgrade on affected distros: - https://git.openembedded.org/openembedded-core/commit/?id=bb7c445004f0706a96192d97bba1c8776595814b - https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=ae74094f8aeae21dcc2510cd0d854b70cfe2192f |