| Summary: | do_rootfs fails with disc space (dnf) although there's enough space left | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Ramon Fried <rfried.dev> | ||||||||
| Component: | deployment | Assignee: | Ramon Fried <rfried.dev> | ||||||||
| Status: | RESOLVED WORKSFORME | QA Contact: | |||||||||
| Severity: | major | ||||||||||
| Priority: | Medium | CC: | alex.kanavin, henry.bruce, ndec13, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie, ross.burton | ||||||||
| Version: | 2.4 | ||||||||||
| Target Milestone: | 0.0.0 | ||||||||||
| Hardware: | x86 | ||||||||||
| OS: | arm64 | ||||||||||
| Whiteboard: | |||||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||||
| Attachments: |
|
||||||||||
At this point in the build any disk space requirements are on the *build host*. Can you check that your build machine wasn't running out of disk space in the directories that the builds happen in? For example if you're doing builds in a container or tmpfs. Hi. There's enough disc space. and I also tested it on another PC and recreated the issue. I suspect there's some glitch in DNF tool. Error Summary ------------- Disk Requirements: At least 752MB more space needed on the / filesystem. ERROR: rpb-desktop-image-1.0-r0 do_rootfs: Function failed: do_rootfs ERROR: Logfile of failure stored in: /local/mnt/workspace/rocko_build/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/temp/log.do_rootfs.26655 ERROR: Task (/local/mnt/workspace/rocko_build/sources/meta-rpb/recipes-samples/images/rpb-desktop-image.bb:do_rootfs) failed with exit code '1' NOTE: Tasks Summary: Attempted 8108 tasks of which 8091 didn't need to be rerun and 1 failed. NOTE: Writing buildhistory Summary: 1 task failed: /local/mnt/workspace/rocko_build/sources/meta-rpb/recipes-samples/images/rpb-desktop-image.bb:do_rootfs Summary: There were 2 WARNING messages shown. Summary: There were 2 ERROR messages shown, returning a non-zero exit code. prastopc@lx-prastopc:/local/mnt/workspace/rocko_build/build$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.8G 4.0K 7.8G 1% /dev tmpfs 1.6G 1.5M 1.6G 1% /run /dev/dm-0 9.8G 6.6G 2.7G 72% / none 4.0K 0 4.0K 0% /sys/fs/cgroup tmpfs 7.8G 1.9M 7.8G 1% /tmp none 5.0M 0 5.0M 0% /run/lock none 7.8G 240K 7.8G 1% /run/shm none 100M 68K 100M 1% /run/user /dev/sda1 464M 112M 328M 26% /boot /dev/mapper/vg00-lv_usr_vice_cache 1.5G 1.1G 350M 75% /usr/vice/cache /dev/mapper/vg00-varvol 4.0G 1.5G 2.3G 39% /var /dev/mapper/vg00-lv_local_mnt 886G 426G 416G 51% /local/mnt 10.18.45.232:/rothschild/pint_0012/scripts 126G 89G 38G 71% /prj/scripts 10.18.44.132:/rothschild/bulk_0020/users/nx 6.0T 5.4T 579G 91% /usr2/nx AFS 2.0T 0 2.0T 0% /afs 10.18.45.229:/rothschild/bulk_0020/users/prastopc 6.0T 5.4T 579G 91% /usr2/prastopc I think I understand the problem now. It appears that NFS mounts cause the issue. If I disable the disk space validation the build passes correctly. You can do it by: defining diskspacecheck=0 or by changing the DNF code directly. I also found reference to the exact problem somewhere in the older packager. http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python-smartpm/smart-add-for-rpm-ignoresize-check.patch?id=yocto-2.0.1 Ross, It appears that your question has been answered, if not please restate the question and put it back in NEEDINFO. Reassigning to Alex as he's the RPM/DNF expert. Alex, can you have a look at this? I'm still unsure why the disk space check fails for you, and I've never seen it happen here, or on our autobuilder, or reported by anyone else. If I can't reproduce, it becomes very hard to figure out what is going on. Also, /local/mnt does not seem to be an NFS mount, or is it? Can I see the output of 'mount' command? You need to dig into rpm source code and find out the precise point where it fails, and what is the system/library call rpm is using to perform the check. Hi, mount output can be seen below. $ mount /dev/mapper/vg00-rotvol on / type ext4 (rw,errors=remount-ro) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/cgroup type tmpfs (rw) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) udev on /dev type devtmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=32G) tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev) none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755) none on /sys/fs/pstore type pstore (rw) /dev/sda1 on /boot type ext3 (rw) /dev/mapper/vg00-varvol on /var type ext4 (rw) /dev/mapper/vg00-lv_usr_vice_cache on /usr/vice/cache type ext4 (rw) /dev/mapper/vg00-lv_local_mnt on /local/mnt type ext4 (rw) rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw) systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd) gvfsd-fuse on /run/user/112/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,user=lightdm) AFS on /afs type afs (rw) /dev/sdb1 on /local2 type ext4 (rw) nfsd on /proc/fs/nfsd type nfsd (rw) 10.18.45.228:/rothschild/bulk_0020/users/rfried on /usr2/rfried type nfs (rw,noquota,sloppy,addr=10.18.45.228) /dev/sdc1 on /usr2/rfried/1 type vfat (rw) 10.18.44.131:/rothschild/bulk_0020/users/nx on /usr2/nx type nfs (rw,noquota,sloppy,addr=10.18.44.131) Alex, It appears Ramon has given you an answer, if you have more questions please put it back into NEEDINFO. Can you add ROOTFS_RPM_DEBUG="1" to your local config and re-run the image creation step? Then I'd need to set the do_rootfs log of that, which should, among other things, have the specific results of available space check on all the filesystems that rpm performs. Also, does this issue happen on a plain poky checkout with no other layers configured? Created attachment 4176 [details]
do_rootfs log with ROOTFS_RPM_DEBUG="1"
(In reply to comment #9) > Can you add ROOTFS_RPM_DEBUG="1" to your local config and re-run the image > creation step? Then I'd need to set the do_rootfs log of that, which should, > among other things, have the specific results of available space check on > all the filesystems that rpm performs. > Looks like the available space check is not printed in the log even though ROOTFS_RPM_DEBUG is set to 1 > Also, does this issue happen on a plain poky checkout with no other layers > configured? no it does not, actually if I remove a big package such as Chromium for instance the problem resolves. Created attachment 4177 [details]
rootfs log with debugging enabled
The log looks exactly like the first one does, e.g. without any debug lines. Did you rebuild the image after making the change? You need to do a 'bitbake -c cleansstate image', or otherwise bitbake won't actually rebuild the image.
Here's for reference, how the do_rootfs log should look like with the debugging enabled.
It appears that it actually runs it with debug : (--rpmverbosity=debug ) Look at the line: ERROR: Could not invoke dnf. Command '/local2/workspace/rocko2/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/recipe-sysroot-native/usr/bin/dnf -v --rpmverbosity=debug -y -c /local2/workspace/rocko2/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/rootfs/etc/dnf/dnf.conf --setopt=reposdir=/local2/workspace/rocko2/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/rootfs/etc/yum.repos.d --repofrompath=oe-repo,/local2/workspace/rocko2/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/oe-rootfs-repo --installroot=/local2/workspace/rocko2/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/rootfs --setopt=logdir=/local2/workspace/rocko2/build/tmp-glibc/work/dragonboard_410c-linaro-linux/rpb-desktop-image/1.0-r0/temp -x lxc --nogpgcheck install locale-base-en-us locale-base-en-gb packagegroup-base-extended packagegroup-core-boot base-passwd packagegroup-core-x11 kernel-modules rpm packagegroup-rpb-x11 dnf packagegroup-core-tools-debug run-postinsts packagegroup-core-ssh-openssh psplash shadow packagegroup-rpb' returned 1: Also, in comparison to your log file, it looks the same in the beginning but it doesn't write anything else afterwards which is weird. Neverthless, just to be certain, I cleaned and built again and got the same results. Then I'm out of ideas. If there's no way to trigger this on plain poky then I cannot help you. The disk space check happens in rpm/lib/transaction.c, particularly in rpmtsCheckDSIProblems() function - you need to find a way to make it print the values it's comparing and other things in the data structures to get some kind of lead on where things go wrong. It's entirely possible the layers you use do something weird at rootfs creation time. I will try to do this. I removed a lot of packages so the image will fit, and then I do get the required prints on the do_rootfs.log whenever you need the logs the most it's not available... Very interesting findins. After adding lot's of verbosity in RPM c library. I see that the problem is that the disc space is checked against the filesystem mounted to "/". I'm still debugging why it tries to check that location and not where the actual build output is located. Additional findins: The function: rpmtsCreateDSI() is the once that calculates the space in rpm/lib/transaction.c It appears that it calculates against "/". I suspect that it expected to run on a chrooted filesystem. I didn't find where the chroot is supposed to be done, although I do see a lot of functions that supposedly handle chroot in the sources. Bottom line, dsi->mntPoint == "/" in the function. But you have plenty of space everywhere, and issuing a chroot() does not affect that. That in itself is not the cause of the error. You need to find out where the numbers that are compared against each other are coming from and why the comparison fails. There's not enough space in the root: /dev/dm-0 9.8G 6.6G 2.7G 72% / Only 2.7G left, the rootfs is bigger than that. Right. So let's agree that it's an upstream rpm issue, and so should be filed to red hat's bugzilla I don't quite get it... While it *might* be an upstream issue, why do we close this bug? it does impact OE/bitbake users if i understand the problem correctly. Basically not having enough space on '/' should be totally transparent to bitbake/OE if we are building on a driver which has enough space. i am not fully convinced it's an upstream bug. rpm is not meant/designed to be used like we do and maybe we are misusing it. i am surprised we close this one so quickly. would it help if we reproduce the issue with a oe-core only build (or a YP build)? @RP: what do you think? I do not think we are misusing rpm here. "--installroot" is a fully supported option in dnf: http://dnf.readthedocs.io/en/latest/command_ref.html and "--root" is a fully supported option in rpm: https://linux.die.net/man/8/rpm Perhaps you can try to reproduce it outside of oe-core altogether: take a bunch of rpms from deploy dir and try to force install them with a host rpm into somewhere on the /local/mnt using --root. Even if it is an upstream issue, this is the sort of thing we chase and fix. I have installed a noarch rpm file to two different partitions (using -r) with debugging enabled (run rpm with -v), and rpm does seem to determine them correctly. Here's what it prints in the two cases: Preparing... D: computing file dispositions D: 0x0000fe01 4096 62904791 20404136 / Preparing... D: computing file dispositions D: 0x00000811 4096 71084402 29090716 / This is of course a bit cryptic and confusing, but here's how you decipher: 1. st_dev from stat(2) 2. block size 3. available blocks 4. available inodes 5. mount point (not sure why it's / in both cases, but the important thing, the available space and device id do differ and correspond to available space on my partitions, so that part works correctly) So, somehow, this breaks down in your installation. Can you get these numbers out? As far as I see, rpm simply calls stat() to get them. Can you send me the exact command you used ? I will try it on my setup. I ran this as root from rpm's upstream source checkout, you need to tweak as appropriate (and copy rpmrc where rpm asks you beforehand): root@kanavin-desktop:/home/ak/development/rpm# RPM_CONFIGDIR=. ./rpm -v -r /home/ak/development/rpm/testdir/ -ivh ../poky/build/tmp/deploy/rpm/noarch/os-release-1.0-r0.noarch.rpm I hit the same issue and it appears that comment 17 explains the problem. rpm looks for space in /, not the filesystem where rootfs is being created. I cleared out space in / (more that rpm said it needed) and rootfs task completed OK. Note that I tested by running a standalone rpm. There might be something happening in the context of dnf where the available space is indeed determined incorrectly. I'll retry again by setting the build directory to various partitions, building images and see how that affects things. Just checked making images with poky master, and ROOTFS_RPM_DEBUG = "1" in local.conf. It still works as expected. When making an image on a separate partition, /media/disk2/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs contains: D: 0x00000811 4096 52318265 28926623 /media/disk2/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs/ and when making an image on the / partition, /home/ak/development/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs contains: D: 0x0000fe01 4096 62100612 20777609 /home/ak/development/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs/ So the partitions are mapped correctly, and disk space is also determined correctly for both. Can you run the same (poky master) and attach the logs here please? No answer, will leave as unassigned. Hi Ramon Fried, Can you still reproduce with the latest repo? If so, could you help to provide the detailed reproduce steps as I cannot reproduce it even there is little rootfs space left and many nfs mounts? # df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 1.6M 2.0G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/mapper/fedora_localhost--live-root 49G 46G 1.5G 97% / tmpfs 2.0G 36K 2.0G 1% /tmp /dev/sda1 976M 221M 689M 25% /boot /dev/mapper/fedora_localhost--live-home 45G 109M 42G 1% /home tmpfs 395M 16K 395M 1% /run/user/42 tmpfs 395M 28K 395M 1% /run/user/1000 tmpfs 395M 4.0K 395M 1% /run/user/1001 /dev/sdb1 196G 58G 129G 31% /buildarea 192.168.5.2:/builddir/test_0515_nfstest1 3.6T 3.3T 190G 95% /mybuild/workspace1 192.168.5.2:/builddir/test_0515_nfstest 3.6T 3.3T 190G 95% /mybuild/workspace2 192.168.5.2:/builddir/test_0409_nettle 3.6T 3.3T 190G 95% /mybuild/workspace3 192.168.5.2:/builddir/test_0510_oeqa 3.6T 3.3T 190G 95% /mybuild/workspace4 And I still succeeded to build the project under /buildarea Ramon, Please reply to Mingli. (In reply to comment #32) > Ramon, Please reply to Mingli. hmm. let me try to reach out to Ramon. He might not be using this email address.. Hi. Sorry for not responding, indeed I'm not working anymore with the registered email address. (I Change it now) Unfortunately, I'm no longer working with this setup, and it's practically impossible for me to recreate it. Sorry I can't assist anymore. If it doesn't reproduce, maybe you should just archive it, until someone else experience the same. hopefully google dives here. Thanks, Ramon. Mingli wasn't able to reproduce the error so marking as resolved. If you see this problem, carefully explain how to recreate it and reopen the defect. Thanks everyone. -- Randy |
Created attachment 4160 [details] do_rootfs log Preconditions/Environment ------------------------- Bitbake 1.36 Yocto Rocko Triggering Action/Cause ----------------------- Running rpb-desktop-image (DB410 target) Expectation ----------- Image created Actual Result ------------- build fails in do_rootfs task: " Error Summary ------------- Disk Requirements: At least 243MB more space needed on the / filesystem. ERROR: rpb-desktop-image-1.0-r0 do_rootfs: Function failed: do_rootfs " Reproducibility --------------- yes. every time.