Bug 10423

Summary: PKGSIZE double counts hard linked files
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Len Sorensen <lsorense>
Component: coreAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact: David Lopez Barriba <davidx.lopez.barriba>
Severity: normal    
Priority: Medium+ CC: jose.perez.carranza, joshuagloe, meta.mr.watcher, meta.watcher
Version: unspecified   
Target Milestone: 2.3 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Len Sorensen 2016-10-12 19:41:20 UTC 
The fix to bug #5334 broke calculations of the package size for any package containing hard links.  It now counts each instance towards the total, while before the change it counted them correctly (since du takes care of that).

Having hard linked files over counted makes it rather hard to know which packages to bother looking at when trying to reduce the size of an image since it will claim a package is using a lot of space when in fact it isn't because of the the hard links.
Comment 1 Len Sorensen 2016-10-12 19:43:11 UTC 
A simple fix is to revert most of the fix for Bug 5334 (leaving the change in scripts/oe-pkgdata-util) and changing the du command from 'du -sk' to 'du -sb'.  The results are then accurate and account for hard links.
Comment 2 Ross Burton 2016-12-15 12:13:04 UTC 
$ ls -l
total 8
-rw-r--r-- 2 ross ross 10 Dec 15 12:10 hard-link-to-test
-rw-r--r-- 2 ross ross 10 Dec 15 12:10 test

$ du -sb .
4106	.

Using du still doesn't solve the problem that PKGSIZE was dependent on the file system type and directory overhead.
Comment 3 Len Sorensen 2016-12-16 21:31:32 UTC 
Yes it does, since -b works in bytes rather than block size, and also implies --apparent-size which means it uses the file size rather than the space used in the filesystem.

The fact it adds 4k for a directory is a bit of a problem perhaps but it is still way more accurate than the current code that double counts hard links.  Certainly in all the packages I looked at, the result was essentially the same as the current code except for the packages with hard links where it was much more accurate.  Most packages the noise from directories is dwarfed by the files.
Comment 4 Len Sorensen 2016-12-16 21:45:17 UTC 
If you want to skip the directories, this seems to work for giving exactly the right size:

find <directory> -type f -print0 | du -sb --files0-from=- | awk '{s+=$1} END {printf "%.0f\n", s}'
Comment 5 Ross Burton 2016-12-21 12:08:49 UTC 
Merged in oe-core e82a7f879419828b42b5cc71e6229c8235090f21.
Comment 6 Len Sorensen 2016-12-21 14:21:51 UTC 
Thanks.  I like the solution.  Nice and clean.