We have a set of autobuilder failures: https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1637 https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1638 https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1639 where a service fails to run at boot. The issue is that /lib/systemd/systemd-remount-fs does not have its executable bit set. pokybuild@ubuntu1804-ty-2:~/yocto-worker/qa-extras2/build/build/tmp/work/core2-64-poky-linux/systemd/1_244.3-r0/package/lib/systemd$ ls -la total 7768 drwxr-xr-x 10 pokybuild pokybuild 4096 Feb 22 13:23 . drwxr-xr-x 6 pokybuild pokybuild 4096 Feb 22 13:23 .. drwxr-xr-x 2 pokybuild pokybuild 4096 Feb 22 13:23 .debug -rwxr-xr-x 2 pokybuild pokybuild 2208952 Feb 22 13:23 libsystemd-shared-244.so drwxr-xr-x 2 pokybuild pokybuild 4096 Feb 22 13:23 network drwxr-xr-x 2 pokybuild pokybuild 4096 Feb 22 13:23 ntp-units.d -rw-r--r-- 2 pokybuild pokybuild 692 Feb 22 12:37 resolv.conf drwxr-xr-x 19 pokybuild pokybuild 12288 Feb 22 13:23 system -rwxr-xr-x 2 pokybuild pokybuild 1562680 Feb 22 13:23 systemd -rw-r--r-- 2 pokybuild pokybuild 33616 Feb 22 13:23 systemd-ac-power -rwxr-xr-x 2 pokybuild pokybuild 22488 Feb 22 13:23 systemd-backlight -rwxr-xr-x 2 pokybuild pokybuild 14368 Feb 22 13:23 systemd-binfmt -rwxr-xr-x 2 pokybuild pokybuild 14384 Feb 22 13:23 systemd-boot-check-no-failures -rwxr-xr-x 2 pokybuild pokybuild 14296 Feb 22 13:23 systemd-cgroups-agent -rwxr-xr-x 2 pokybuild pokybuild 18576 Feb 22 13:23 systemd-dissect -rwxr-xr-x 2 pokybuild pokybuild 26584 Feb 22 13:23 systemd-fsck -rwxr-xr-x 2 pokybuild pokybuild 14368 Feb 22 13:23 systemd-growfs -rwxr-xr-x 2 pokybuild pokybuild 14304 Feb 22 13:23 systemd-hibernate-resume -rwxr-xr-x 2 pokybuild pokybuild 30760 Feb 22 13:23 systemd-hostnamed -rw-r--r-- 2 pokybuild pokybuild 54944 Feb 22 13:23 systemd-initctl -rwxr-xr-x 2 pokybuild pokybuild 153656 Feb 22 13:23 systemd-journald -rwxr-xr-x 2 pokybuild pokybuild 47136 Feb 22 13:23 systemd-localed -rwxr-xr-x 2 pokybuild pokybuild 260200 Feb 22 13:23 systemd-logind -rw-r--r-- 2 pokybuild pokybuild 525056 Feb 22 13:23 systemd-machined -rwxr-xr-x 2 pokybuild pokybuild 14288 Feb 22 13:23 systemd-makefs -rwxr-xr-x 2 pokybuild pokybuild 18576 Feb 22 13:23 systemd-modules-load -rwxr-xr-x 2 pokybuild pokybuild 891184 Feb 22 13:23 systemd-networkd -rwxr-xr-x 2 pokybuild pokybuild 30904 Feb 22 13:23 systemd-networkd-wait-online -rwxr-xr-x 2 pokybuild pokybuild 34864 Feb 22 13:23 systemd-network-generator -rwxr-xr-x 2 pokybuild pokybuild 22584 Feb 22 13:23 systemd-pstore -rwxr-xr-x 2 pokybuild pokybuild 14376 Feb 22 13:23 systemd-quotacheck -rwxr-xr-x 2 pokybuild pokybuild 18392 Feb 22 13:23 systemd-random-seed -rw-r--r-- 2 pokybuild pokybuild 80792 Feb 22 13:23 systemd-remount-fs -rwxr-xr-x 2 pokybuild pokybuild 14296 Feb 22 13:23 systemd-reply-password -rwxr-xr-x 2 pokybuild pokybuild 399400 Feb 22 13:23 systemd-resolved -rwxr-xr-x 2 pokybuild pokybuild 22560 Feb 22 13:23 systemd-rfkill -rwxr-xr-x 2 pokybuild pokybuild 51248 Feb 22 13:23 systemd-shutdown -rwxr-xr-x 2 pokybuild pokybuild 26760 Feb 22 13:23 systemd-sleep -rwxr-xr-x 2 pokybuild pokybuild 30768 Feb 22 13:23 systemd-socket-proxyd -rwxr-xr-x 2 pokybuild pokybuild 14296 Feb 22 13:23 systemd-sulogin-shell -rwxr-xr-x 2 pokybuild pokybuild 22664 Feb 22 13:23 systemd-sysctl -rwxr-xr-x 2 pokybuild pokybuild 1258 Feb 22 13:23 systemd-sysv-install -rw-r--r-- 2 pokybuild pokybuild 148824 Feb 22 13:23 systemd-timedated -rwxr-xr-x 2 pokybuild pokybuild 55336 Feb 22 13:23 systemd-timesyncd -rwxr-xr-x 2 pokybuild pokybuild 14296 Feb 22 13:23 systemd-time-wait-sync -rwxr-xr-x 2 pokybuild pokybuild 784920 Feb 22 13:23 systemd-udevd -rwxr-xr-x 2 pokybuild pokybuild 14296 Feb 22 13:23 systemd-update-done -rwxr-xr-x 2 pokybuild pokybuild 14376 Feb 22 13:23 systemd-update-utmp -rwxr-xr-x 2 pokybuild pokybuild 18400 Feb 22 13:23 systemd-user-runtime-dir -rwxr-xr-x 2 pokybuild pokybuild 18392 Feb 22 13:23 systemd-vconsole-setup -rwxr-xr-x 2 pokybuild pokybuild 14296 Feb 22 13:23 systemd-volatile-root drwxr-xr-x 3 pokybuild pokybuild 4096 Feb 22 13:23 system-generators drwxr-xr-x 2 pokybuild pokybuild 4096 Feb 22 13:23 system-preset drwxr-xr-x 2 pokybuild pokybuild 4096 Feb 22 13:23 system-shutdown drwxr-xr-x 2 pokybuild pokybuild 4096 Feb 22 13:23 system-sleep There are a few other files in that directory that aren't executable and probably should be. I've tracked this back to a do_package sstate artefact. We don't have the original build that created this unfortunately but its clearly something that went wrong in the build of systemd.
I appreciate I listed this outside pseudo but it doesn't matter, the broken permissions are the same inside pseudo.
I believe it was built on centos7-ty-1 but trying a build there didn't result in the same permissions issue. Had to resort to a cleansstate on systemd on the infrastructure to clear the corrupt sstate out. Still no idea how this got in :(
The plan is to see if this happens again. If anyone has seen this sort of file permissions failure, please comment.
I check with people who know the WR testsuite and results and while we don't have a specific test, we do have test code that would fail if this problem occurred so we are almost certainly not seeing the issue. We will add a specific test that would catch this error in the coming month or so.
We'll see if this happens again during 3.2 but if not, I'm tempted to close it as can't reproduce.
Something similar happened to me today while building an image. gio-querymodules from glib-2.0 didn't have executable bit set which resulted in qemu failing with a permissions denied error while generating rootfs. Doing -c install -f glib-2.0 also didn't help. The build directory had the correct permissions set but the image directory didn't. Meson logs didn't show anything helpful either. Forcing configure to be re-run and then install helped. Any suggestions on how to debug?
This is likely fixed by the changes made to oe-core to: 1. reduce the number of directories/files that pseudo manages cba4f51f01 pseudo: Upgrade to include mkostemp64 wrapper ea22fb3aa1 pseudo: Update to account for patches merged on branch 016ee90e21 abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes ... f9ae80aee9 pseudo: Fix statx function usage 13500f5234 wic: Handle new PSEUDO_IGNORE_PATHS variable ad8f5532ff base/bitbake.conf: Enable pseudo path filtering 3006bbe776 pseudo: Add pathfix patch e1e481f360 pseudo: Add may unlink patch 929a27bf6c pseudo: Fix xattr segfault 9ce621fa20 psuedo: Add tracking of linked files for fds 2db491d97d pseudo: Abort on mismatch patch 8defb687a2 pseudo: Add support for ignoring paths from the pseudo DB 1c13149b81 pseudo: Ignore mismatched inodes from the db and 2. change associated pseudo warnings into errors. https://git.openembedded.org/openembedded-core/commit/?id=2db491d97da08d44ebd257f98489550a82a7935c I'm going to leave the bug open until we review it and confirm that the error hasn't happened since these changes in late September.
A similar failure happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/3070/steps/24/logs/stdio
(In reply to comment #8) > A similar failure happened again: > https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/3070/steps/ > 24/logs/stdio This looks different from the permissions problem that this bug is for.
Anuj, I agree that: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13810#c8 is a different issue. I'm raising this bug to a M+ for 3.3-M3 so that we can discuss it in the coming weeks and hopefully agree that the changes to pseudo paths made late last year seem to have fixed this issue. I don't thing the same issue has happened in the last few months but since it's a rare error, I'm not sure how long people want to leave the bug opened to look for it.
I don't think we've seen this since the series of patches mentioned earlier including: https://git.openembedded.org/openembedded-core/commit/?id=2db491d97d were merged. Please re-open if you still see this happening on master or 3.3 otherwise. Added Richard Purdie and Steve Sakoman to CC to give them a chance to object.