Bug 13810 - systemd file executable bits disappearing
Summary: systemd file executable bits disappearing
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.0
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.3 M3
Assignee: Randy MacLeod
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-02-22 17:05 UTC by Richard Purdie
Modified: 2021-03-16 19:04 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2020-02-22 17:05:49 UTC
We have a set of autobuilder failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1637
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1638
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1639

where a service fails to run at boot. The issue is that /lib/systemd/systemd-remount-fs does not have its executable bit set.

pokybuild@ubuntu1804-ty-2:~/yocto-worker/qa-extras2/build/build/tmp/work/core2-64-poky-linux/systemd/1_244.3-r0/package/lib/systemd$ ls -la
total 7768
drwxr-xr-x 10 pokybuild pokybuild    4096 Feb 22 13:23 .
drwxr-xr-x  6 pokybuild pokybuild    4096 Feb 22 13:23 ..
drwxr-xr-x  2 pokybuild pokybuild    4096 Feb 22 13:23 .debug
-rwxr-xr-x  2 pokybuild pokybuild 2208952 Feb 22 13:23 libsystemd-shared-244.so
drwxr-xr-x  2 pokybuild pokybuild    4096 Feb 22 13:23 network
drwxr-xr-x  2 pokybuild pokybuild    4096 Feb 22 13:23 ntp-units.d
-rw-r--r--  2 pokybuild pokybuild     692 Feb 22 12:37 resolv.conf
drwxr-xr-x 19 pokybuild pokybuild   12288 Feb 22 13:23 system
-rwxr-xr-x  2 pokybuild pokybuild 1562680 Feb 22 13:23 systemd
-rw-r--r--  2 pokybuild pokybuild   33616 Feb 22 13:23 systemd-ac-power
-rwxr-xr-x  2 pokybuild pokybuild   22488 Feb 22 13:23 systemd-backlight
-rwxr-xr-x  2 pokybuild pokybuild   14368 Feb 22 13:23 systemd-binfmt
-rwxr-xr-x  2 pokybuild pokybuild   14384 Feb 22 13:23 systemd-boot-check-no-failures
-rwxr-xr-x  2 pokybuild pokybuild   14296 Feb 22 13:23 systemd-cgroups-agent
-rwxr-xr-x  2 pokybuild pokybuild   18576 Feb 22 13:23 systemd-dissect
-rwxr-xr-x  2 pokybuild pokybuild   26584 Feb 22 13:23 systemd-fsck
-rwxr-xr-x  2 pokybuild pokybuild   14368 Feb 22 13:23 systemd-growfs
-rwxr-xr-x  2 pokybuild pokybuild   14304 Feb 22 13:23 systemd-hibernate-resume
-rwxr-xr-x  2 pokybuild pokybuild   30760 Feb 22 13:23 systemd-hostnamed
-rw-r--r--  2 pokybuild pokybuild   54944 Feb 22 13:23 systemd-initctl
-rwxr-xr-x  2 pokybuild pokybuild  153656 Feb 22 13:23 systemd-journald
-rwxr-xr-x  2 pokybuild pokybuild   47136 Feb 22 13:23 systemd-localed
-rwxr-xr-x  2 pokybuild pokybuild  260200 Feb 22 13:23 systemd-logind
-rw-r--r--  2 pokybuild pokybuild  525056 Feb 22 13:23 systemd-machined
-rwxr-xr-x  2 pokybuild pokybuild   14288 Feb 22 13:23 systemd-makefs
-rwxr-xr-x  2 pokybuild pokybuild   18576 Feb 22 13:23 systemd-modules-load
-rwxr-xr-x  2 pokybuild pokybuild  891184 Feb 22 13:23 systemd-networkd
-rwxr-xr-x  2 pokybuild pokybuild   30904 Feb 22 13:23 systemd-networkd-wait-online
-rwxr-xr-x  2 pokybuild pokybuild   34864 Feb 22 13:23 systemd-network-generator
-rwxr-xr-x  2 pokybuild pokybuild   22584 Feb 22 13:23 systemd-pstore
-rwxr-xr-x  2 pokybuild pokybuild   14376 Feb 22 13:23 systemd-quotacheck
-rwxr-xr-x  2 pokybuild pokybuild   18392 Feb 22 13:23 systemd-random-seed
-rw-r--r--  2 pokybuild pokybuild   80792 Feb 22 13:23 systemd-remount-fs
-rwxr-xr-x  2 pokybuild pokybuild   14296 Feb 22 13:23 systemd-reply-password
-rwxr-xr-x  2 pokybuild pokybuild  399400 Feb 22 13:23 systemd-resolved
-rwxr-xr-x  2 pokybuild pokybuild   22560 Feb 22 13:23 systemd-rfkill
-rwxr-xr-x  2 pokybuild pokybuild   51248 Feb 22 13:23 systemd-shutdown
-rwxr-xr-x  2 pokybuild pokybuild   26760 Feb 22 13:23 systemd-sleep
-rwxr-xr-x  2 pokybuild pokybuild   30768 Feb 22 13:23 systemd-socket-proxyd
-rwxr-xr-x  2 pokybuild pokybuild   14296 Feb 22 13:23 systemd-sulogin-shell
-rwxr-xr-x  2 pokybuild pokybuild   22664 Feb 22 13:23 systemd-sysctl
-rwxr-xr-x  2 pokybuild pokybuild    1258 Feb 22 13:23 systemd-sysv-install
-rw-r--r--  2 pokybuild pokybuild  148824 Feb 22 13:23 systemd-timedated
-rwxr-xr-x  2 pokybuild pokybuild   55336 Feb 22 13:23 systemd-timesyncd
-rwxr-xr-x  2 pokybuild pokybuild   14296 Feb 22 13:23 systemd-time-wait-sync
-rwxr-xr-x  2 pokybuild pokybuild  784920 Feb 22 13:23 systemd-udevd
-rwxr-xr-x  2 pokybuild pokybuild   14296 Feb 22 13:23 systemd-update-done
-rwxr-xr-x  2 pokybuild pokybuild   14376 Feb 22 13:23 systemd-update-utmp
-rwxr-xr-x  2 pokybuild pokybuild   18400 Feb 22 13:23 systemd-user-runtime-dir
-rwxr-xr-x  2 pokybuild pokybuild   18392 Feb 22 13:23 systemd-vconsole-setup
-rwxr-xr-x  2 pokybuild pokybuild   14296 Feb 22 13:23 systemd-volatile-root
drwxr-xr-x  3 pokybuild pokybuild    4096 Feb 22 13:23 system-generators
drwxr-xr-x  2 pokybuild pokybuild    4096 Feb 22 13:23 system-preset
drwxr-xr-x  2 pokybuild pokybuild    4096 Feb 22 13:23 system-shutdown
drwxr-xr-x  2 pokybuild pokybuild    4096 Feb 22 13:23 system-sleep

There are a few other files in that directory that aren't executable and probably should be. I've tracked this back to a do_package sstate artefact. We don't have the original build that created this unfortunately but its clearly something that went wrong in the build of systemd.
Comment 1 Richard Purdie 2020-02-22 17:06:30 UTC
I appreciate I listed this outside pseudo but it doesn't matter, the broken permissions are the same inside pseudo.
Comment 2 Richard Purdie 2020-02-22 17:25:31 UTC
I believe it was built on centos7-ty-1 but trying a build there didn't result in the same permissions issue. Had to resort to a cleansstate on systemd on the infrastructure to clear the corrupt sstate out. Still no idea how this got in :(
Comment 3 Randy MacLeod 2020-02-27 16:27:08 UTC
The plan is to see if this happens again.
If anyone has seen this sort of file permissions failure, please comment.
Comment 4 Randy MacLeod 2020-02-28 18:32:50 UTC
I check with people who know the WR testsuite and results and while we don't have a specific test, we do have test code that would fail if this problem occurred so we are almost certainly not seeing the issue.

We will add a specific test that would catch this error in the coming month or so.
Comment 5 Randy MacLeod 2020-04-13 12:44:34 UTC
We'll see if this happens again during 3.2 but if not, I'm tempted to close it as can't reproduce.
Comment 6 Anuj Mittal 2020-05-14 17:30:17 UTC
Something similar happened to me today while building an image. gio-querymodules from glib-2.0 didn't have executable bit set which resulted in qemu failing with a permissions denied error while generating rootfs.

Doing -c install -f glib-2.0 also didn't help. The build directory had the correct permissions set but the image directory didn't. Meson logs didn't show anything helpful either.

Forcing configure to be re-run and then install helped. Any suggestions on how to debug?
Comment 7 Randy MacLeod 2020-11-03 10:45:13 UTC
This is likely fixed by the changes made to oe-core to:
 1. reduce the number of directories/files that pseudo manages 
cba4f51f01 pseudo: Upgrade to include mkostemp64 wrapper
ea22fb3aa1 pseudo: Update to account for patches merged on branch
016ee90e21 abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes
...

f9ae80aee9 pseudo: Fix statx function usage
13500f5234 wic: Handle new PSEUDO_IGNORE_PATHS variable
ad8f5532ff base/bitbake.conf: Enable pseudo path filtering
3006bbe776 pseudo: Add pathfix patch
e1e481f360 pseudo: Add may unlink patch
929a27bf6c pseudo: Fix xattr segfault
9ce621fa20 psuedo: Add tracking of linked files for fds
2db491d97d pseudo: Abort on mismatch patch
8defb687a2 pseudo: Add support for ignoring paths from the pseudo DB
1c13149b81 pseudo: Ignore mismatched inodes from the db


and

 2. change associated pseudo warnings into errors.

https://git.openembedded.org/openembedded-core/commit/?id=2db491d97da08d44ebd257f98489550a82a7935c

I'm going to leave the bug open until we review it and confirm that the error hasn't happened since these changes in late September.
Comment 8 Minjae Kim 2021-02-10 14:39:36 UTC
A similar failure happened again:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/3070/steps/24/logs/stdio
Comment 9 Anuj Mittal 2021-02-10 16:22:09 UTC
(In reply to comment #8)
> A similar failure happened again:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/3070/steps/
> 24/logs/stdio

This looks different from the permissions problem that this bug is for.
Comment 10 Randy MacLeod 2021-02-11 16:54:36 UTC
Anuj, I agree that:
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=13810#c8
is a different issue. I'm raising this bug to a M+ for 3.3-M3 so that we can 
discuss it in the coming weeks and hopefully agree that the changes to pseudo paths made late last year seem to have fixed this issue. I don't thing the same issue has happened in the last few months but since it's a rare error, I'm not sure how long people want to leave the bug opened to look for it.
Comment 11 Randy MacLeod 2021-03-16 19:04:10 UTC
I don't think we've seen this since the series of patches mentioned earlier including:
  https://git.openembedded.org/openembedded-core/commit/?id=2db491d97d
were merged. Please re-open if you still see this happening on master or 3.3 otherwise.

Added Richard Purdie and Steve Sakoman to CC to give them a chance to object.