The 'du' invokation in Image::_get_rootfs_size inaccurately assumes that the local filesystem onto which the do_rootfs has layed out the rootfs files will report the correct size used subsequently by the image file creation allocation. This yields in a possibly smaller image file being allocated which will not be large enough to fit all rootfs files. Basically it ties the host working directory filesystem with the target image filesystem. Different filesystems will exhibit variants of this symptom. My typical scenario involves hosting the work directory on a zfs dataset which happens to do lz4 compression on the fly. In this case the reported size is vastly different when invoking du with '--apparent-size' /builds/yocto/i7/tmp-glibc/work/intel_corei7_64-oe-linux/qt4e-demo-image/1.0-r3/rootfs$ du -ks . 278697 . /builds/yocto/i7/tmp-glibc/work/intel_corei7_64-oe-linux/qt4e-demo-image/1.0-r3/rootfs$ du -ks --apparent-size . 359343 . This simple patch resolves this and has proven quite viable on my environment for years (was in image_types.bbclass before): diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py index 0ce303d..2edbe5e 100644 --- a/meta/lib/oe/image.py +++ b/meta/lib/oe/image.py @@ -160,7 +160,7 @@ class Image(ImageDepGraph): rootfs_extra_space = eval(self.d.getVar('IMAGE_ROOTFS_EXTRA_SPACE', True)) rootfs_maxsize = self.d.getVar('IMAGE_ROOTFS_MAXSIZE', True) - output = subprocess.check_output(['du', '-ks', + output = subprocess.check_output(['du', '-ks --apparent-size', self.d.getVar('IMAGE_ROOTFS', True)]) size_kb = int(output.split()[0]) base_size = size_kb * overhead_factor
sorry, that's: diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py index 0ce303d..2edbe5e 100644 --- a/meta/lib/oe/image.py +++ b/meta/lib/oe/image.py @@ -160,7 +160,7 @@ class Image(ImageDepGraph): rootfs_extra_space = eval(self.d.getVar('IMAGE_ROOTFS_EXTRA_SPACE', True)) rootfs_maxsize = self.d.getVar('IMAGE_ROOTFS_MAXSIZE', True) - output = subprocess.check_output(['du', '-ks', + output = subprocess.check_output(['du', '-ks', '--apparent-size', self.d.getVar('IMAGE_ROOTFS', True)]) size_kb = int(output.split()[0]) base_size = size_kb * overhead_factor
We usually ask for patches on the mailing list formatted as git patches, but this is so simple I did it myself.
I applied the patch but all of the sanity tests are now failing with errors such as: | FAIL: test_df (oeqa.runtime.df.DfTest) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm/build/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f | return func(*args) | File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm/build/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f | return f(*args) | File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-rpm/build/meta/lib/oeqa/runtime/df.py", line 12, in test_df | self.assertTrue(int(output)>5120, msg="Not enough space on image. Current size is %s" % output) | AssertionError: False is not true : Not enough space on image. Current size is 0 With this change, the image doesn't have any slack space. I'll look at this after release.
From du's man --apparent-size print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like ======= So when used, the block size of the fs is not taken into account. It makes sense then that the total size reported would be smaller. I'm surprised we never had any issues with this... yet. The use of du without apparent-size is then only valid if the both work dir and target image are the same and built using the same block size. It's clear more thinking needs to be put into this.
I've made a new incarnation of my patch, here it is as a RFC: diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py index 0ce303d..8a38619 100644 --- a/meta/lib/oe/image.py +++ b/meta/lib/oe/image.py @@ -160,10 +160,15 @@ class Image(ImageDepGraph): rootfs_extra_space = eval(self.d.getVar('IMAGE_ROOTFS_EXTRA_SPACE', True)) rootfs_maxsize = self.d.getVar('IMAGE_ROOTFS_MAXSIZE', True) + output = subprocess.check_output(['du', '-ks', '--apparent-size', + self.d.getVar('IMAGE_ROOTFS', True)]) + apparent_size_kb = int(output.split()[0]) + output = subprocess.check_output(['du', '-ks', self.d.getVar('IMAGE_ROOTFS', True)]) - size_kb = int(output.split()[0]) - base_size = size_kb * overhead_factor + fs_reported_size_kb = int(output.split()[0]) + + base_size = max(fs_reported_size_kb, apparent_size_kb) * overhead_factor base_size = (base_size, rootfs_req_size)[base_size < rootfs_req_size] + \ rootfs_extra_space
I can submit a proper patch on the mailing list if there is interest in this.
A patch on the list would be much appreciated.
Just found this again. I'm not sure I understand why the apparent size isn't always the right thing to use?
(In reply to comment #8) > I'm not sure I understand why the apparent size isn't always the right thing > to use? In cases where the filesystem hosting the TMP_DIR would implement some sort of compression of the underlying storage. My use case is that my TMP_DIR is a zfs dataset with lz4 compression on it. In this case, the du command will report that the "Disk Usage" is less than the actual files would seem. At any rate, using du without "--apparent-size" ties the host TMP_DIR filesystem to the image size. It works right now because when du is used on ext4, it reports a fairly accurate size estimate of the given directory tree. Using apparent-size decouples the TMP_DIR of the image recipe and reports a total sum that more closely represent the stored files. I've been hosting all my builds (yocto and others) on a zfs dataset with compression=lz4, sync=disabled and atime=off for years with great performance and stability. The real gain is not having to provision for a build partition to get the "no sync" behaviour. Cheers!
I understand why you would use du --apparent-size as the same problem happens if the rootfs contains highly sparse files. I don't understand your V2 patch which uses the maximum of the apparent size and the reported size.
(In reply to comment #10) > I understand why you would use du --apparent-size as the same problem > happens if the rootfs contains highly sparse files. I don't understand your > V2 patch which uses the maximum of the apparent size and the reported size. Yeah it was a simple solution to a complex problem (on closer inspection). And I reread your question. My answer only re-asks your question. It seems like there are two extremes here: * sparse files * compressed host filesystem There is no single use of du which satisfies both. I remember thinking about this back then, I thought the only true solution would be to recurse through all the files and invoke du on each using apparent-size or not depending if the file is sparse or not. Added to this, a way to calculate the target filesystem's tree structure overhead would have to be added on top of the final sum. Maybe a simple 1.1x factor or something. Anyway, supporting other host TMP_DIR filesystems does seem like an improvement, but may not align with current OE/yocto development priorities.
The apparent size of a 1GB file that is entirely sparse is 1GB. Surely the apparent size of a file that compresses well on your funky ZFS partition is the size of the uncompressed file. Either way, the apparent size surely the right size to use - the only way the non-apparent size can be larger is if there are large blocks on the file system and then we're surely talking about a fraction of a block size. I must be missing something... maybe it's time to go to bed. :)
(In reply to comment #12) Yes, apparent-size would yield the actual ("un-sparsed") file size. If you were to copy a rootfs containing sparse files onto a filesystem image which doesn't support the feature, you sure would want to use du --apparent-size in order to get the "all features off" size of the rootfs tree. Here are some tests I did on debian 8 (du from coreutils 8.23-4) /builds $ grep " /builds" /proc/mounts mini/builds /builds zfs rw,noatime,xattr,noacl 0 0 /builds$ truncate -s 1m sparsetest /builds$ ls -l sparsetest -rw-r--r-- 1 * * 1048576 Apr 12 08:24 sparsetest /builds$ du -ks sparsetest 1 sparsetest /builds$ du -ks --apparent-size sparsetest 1024 sparsetest ============================================ Right, now on ext4: /builds$ grep " / ext4" /proc/mounts /dev/sda2 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 /testdir$ truncate -s 1m sparsetest /testdir$ ls -l sparsetest -rw-r--r-- 1 * * 1048576 Apr 12 08:34 sparsetest /testdir$ du -ks sparsetest 0 sparsetest /testdir$ du -ks --apparent-size sparsetest 1024 sparsetest =========================================== Doing the same test using zeros (dd if=/dev/zero of=zeros bs=1M count=1). I get the proper size on ext4 using apparent or juste "-ks", but on ZFS: /builds$ du -ks zeros 1 zeros /builds$ du -ks --apparent-size zeros 1024 zeros So there you have it. The last remaining factor is the filesystem overhead. If a rootfs tree has many MANY small files, the overhead will be great, and depend on the block size and filesystem structure. Apparent-size will fail to consider these and will juste make the sum of all the apparent files sizes it encounters. A safety margin added to du's reported size, as is done right now, should make it work in almost all cases. Another alternative would be to: 1. create the target image file "sparsely" (using truncate) with a size 3-4 times the source rootfs apparent-size 2. write the files to it 3. using parted, shrink the target filesystem on the image to a size which takes into account IMAGE_ROOTFS_EXTRA_SPACE 4. using truncate, snuggly shrink the image to the filesystem size. Looking back at the test that had failed with my patch, I wonder if it's not just that the test assumed a certain extra space would be available on the image (mounted RW I assume). The patch changed that extra space just enough to fail the test. If you ask me, I would say the test was fragile and should not make assumptions on free extra space on the file system. OR, express this assumptions in the form of IMAGE_ROOTFS_EXTRA_SPACE or something. My 2 cents. BTW, ZFS is soooo funky, it even supports sparse files! ;)
(In reply to comment #13) sorry, my french autocorrector changed a few "just" to "juste"!
When build with nice options: # nice -n 19 ionice -c 3 bitbake core-image-minimal Flag '--apparent-size' doesn't help avoid problem with incorrect rootfs size. I added command 'sync' before 'du', and all problems disappear.
(In reply to comment #15) > When build with nice options: > > # nice -n 19 ionice -c 3 bitbake core-image-minimal > > Flag '--apparent-size' doesn't help avoid problem with incorrect rootfs size. > > I added command 'sync' before 'du', and all problems disappear. I doubt the sync is actually doing anything... the fact that write caches are actually committed to physical medium or not should never impact the experience of programs inspecting the filesystem. You might have other side effects at work here. Like the fact that when you call 'sync', that task is probably blocked for a while, and other tasks can move ahead. This might mask, but not fix, a race condition. Back on the original topic, I have also found that if we execute du to get the image size for a Fat32 target image with a TMP_DIR on the supported ext4 FS, the number returned by du is off by a good margin. The OVERHEAD_FACTOR saves the day by adding a lot of extra room for the file copy operation to succeed. The main cause is how each FS allocates whole sectors for each files. This will obviously vary quite a lot from FS to FS. The worst part is that it may be subject to local FS configuration, i.e. if the workstation admin made changes in mkfs.ext4 of the TMP_DIR partition. At any rate, using du + OVERHEAD_FACTOR is a hack that still works most of the times, but will not cover 100% of cases. The occasional tweak will be required. The alternative may be too complex for anyone to tackle, especially since tweaking OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE usually does the trick. For the brave souls who might try for ext[234], consider my parted suggestion in comment #13
> I doubt the sync is actually doing anything I have seen some filesystems where the block count seen by du isn't updated until the data is written to disk. The sync would wait for this to happen. Arguably, this is an error in the filesystem implementation, but is probably something we have to live with. I'm not sure why apparent size wouldn't work, but maybe total size is also not updated. I believe this was xfs where I saw this, but it might also be btrfs.
I tested this 'sync' workaround on Build Server, my work PC and PC of colleague. Never see fail after 50+ build. Yocto "morty" branch with Freescale and Boundary layers. Configuration of Build Server: > Ubuntu 14.04 x64, CPU 4 core/Core-i7, 4Gb RAM, kernel 3.13.11, RAID0 1Tb ext4 > parallel make - 4, bb threads - 4 Configuration of work PC: > Ubuntu 16.04 x64, CPU 8 core/Core-i7, 16Gb RAM, kernel 4.12, HDD 0.5Tb btrfs + HDD 0.5Tb ext4 > parallel make - 8, bb threads - 4 Configuration of colleague's PC: > Main system Win7, Linux Mint 17 x64 on VirtualBox, CPU 8 core/Core-i7 (4 core for VirtualBox), 16Gb RAM (8Gb for VirtualBox), kernel 3.13, HDD 0.5Tb ext4 directly mounted to VirtualBox. > parallel make - 4, bb threads - 4 All fs mounted with default options, nothing tuned. All packages builts with success, but on final stage creating image_ext2 yocto randomly crash, when trying install packages, because allocated size for ext2-image time to time smaller whan real size of rootfs. I also added bb.warn for print rootfs size which 'du' get, each time on fail it smaller than real size. Envs for build image in local.conf: > IMAGE_OVERHEAD_FACTOR = "1.05" > IMAGE_ROOTFS_EXTRA_SPACE = "1000" Before workaround with 'sync': > ext4: > - randomly crash on server and my work PC, after 10-15 builds. > - on colleague's PC yocto crash every 5-7 build. > btrfs: > - it crash very often, every 2nd or 3rd build. Envs for build image in local.conf: > IMAGE_ROOTFS_SIZE = "0" > IMAGE_OVERHEAD_FACTOR = "1.0" > IMAGE_ROOTFS_ALIGNMENT = "1" > IMAGE_ROOTFS_EXTRA_SPACE = "0" > ext4: > - randomly crash on server and my work PC, after 5-10 builds. > - on colleague's PC yocto crash every 2nd/3rd build. > btrfs: > - every build is crashed. Then I add '--apparent-size' flag for 'du': > ext4: > - randomly crash on server and my work PC, after 10-15 builds. > - on colleague's PC yocto crash every 5-7 build. > btrfs: > - every build is crashed. flag doesn't help. After workaround with 'sync': Envs for build image in local.conf: > IMAGE_ROOTFS_SIZE = "0" > IMAGE_OVERHEAD_FACTOR = "1.0" > IMAGE_ROOTFS_ALIGNMENT = "1" > IMAGE_ROOTFS_EXTRA_SPACE = "0" > ext4: > - never crashed on all tested systems after 25+ builds, everything is OK. > btrfs: > - never crashed after 25+ builds, everything is OK. > bb.warn show the same roofs size for all builds. Maybe sync add some delay for do some parallel magic job, but it help avoid problem for our team.
Command for test: > while true; do > nice -n 19 ionice -c 3 bitbake core-image-minimal -c cleanall > nice -n 19 ionice -c 3 bitbake core-image-minimal > done
(In reply to comment #19) This is quite concerning... I wonder why we are not seeing this more then. The more I contemplate this, the more I think a custom du function should be created. Something that would find all files within the rootfs subdir, and stat each file individually. For sure, the host filesystem should be taken out of the loop completely to avoid any of these weird effects. Though sync is not such a bad workaround if it definitely fixes all cases.
We started seeing problems like this on the autobuilder so patch sent - again - to add --apparent-size to the du. It appears to do roughly the right thing for directories and is much better when dealing with compressed file systems. Leaving this open for the better solution of creating an image double the size we need and then shrinking it again. For future me, 'debugfs -R stats' outputs enough information to be used to do the calculations before shrinking.
*** Bug 12235 has been marked as a duplicate of this bug. ***
*** Bug 13216 has been marked as a duplicate of this bug. ***
We have this problem inside lxc with a zfs backend. Ross Burton provided us with a WIP branch that did solve the issue. [1] [1] https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/du It seems that this was never merged since it enlarged image size in a certain situation and yielded a regression. See this IRC exchange [2] [2] https://www.yoctoproject.org/irc/latest.log.html#t2019-03-06T14:52:38 Thanks
Hi Ross, Now that we don't have that 4 GB limitation, is this something that should be merged?
*** Bug 12304 has been marked as a duplicate of this bug. ***
Bulk move from 4.99 or 0.00 to 5.99