Bug 10423 - PKGSIZE double counts hard linked files
Summary: PKGSIZE double counts hard linked files
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.3 M1
Assignee: Ross Burton
QA Contact: David Lopez Barriba
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-10-12 19:41 UTC by Len Sorensen
Modified: 2017-01-17 14:07 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Len Sorensen 2016-10-12 19:41:20 UTC
The fix to bug #5334 broke calculations of the package size for any package containing hard links.  It now counts each instance towards the total, while before the change it counted them correctly (since du takes care of that).

Having hard linked files over counted makes it rather hard to know which packages to bother looking at when trying to reduce the size of an image since it will claim a package is using a lot of space when in fact it isn't because of the the hard links.
Comment 1 Len Sorensen 2016-10-12 19:43:11 UTC
A simple fix is to revert most of the fix for Bug 5334 (leaving the change in scripts/oe-pkgdata-util) and changing the du command from 'du -sk' to 'du -sb'.  The results are then accurate and account for hard links.
Comment 2 Ross Burton 2016-12-15 12:13:04 UTC
$ ls -l
total 8
-rw-r--r-- 2 ross ross 10 Dec 15 12:10 hard-link-to-test
-rw-r--r-- 2 ross ross 10 Dec 15 12:10 test

$ du -sb .
4106	.

Using du still doesn't solve the problem that PKGSIZE was dependent on the file system type and directory overhead.
Comment 3 Len Sorensen 2016-12-16 21:31:32 UTC
Yes it does, since -b works in bytes rather than block size, and also implies --apparent-size which means it uses the file size rather than the space used in the filesystem.

The fact it adds 4k for a directory is a bit of a problem perhaps but it is still way more accurate than the current code that double counts hard links.  Certainly in all the packages I looked at, the result was essentially the same as the current code except for the packages with hard links where it was much more accurate.  Most packages the noise from directories is dwarfed by the files.
Comment 4 Len Sorensen 2016-12-16 21:45:17 UTC
If you want to skip the directories, this seems to work for giving exactly the right size:

find <directory> -type f -print0 | du -sb --files0-from=- | awk '{s+=$1} END {printf "%.0f\n", s}'
Comment 5 Ross Burton 2016-12-21 12:08:49 UTC
Merged in oe-core e82a7f879419828b42b5cc71e6229c8235090f21.
Comment 6 Len Sorensen 2016-12-21 14:21:51 UTC
Thanks.  I like the solution.  Nice and clean.