Bug 7586

Summary:	Hardlinks are not preserved and files are not stripped
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Enrico Scholz <enrico.scholz>
Component:	core	Assignee:	Ed Bartosh <eduard.bartosh>
Status:	VERIFIED FIXED	QA Contact:	Costin <costin.c.constantin>
Severity:	normal
Priority:	Medium+	CC:	alexandru.c.georgescu, bluelightning, mark.hatle, Martin.Jansa, meta.mr.watcher, meta.watcher
Version:	unspecified
Target Milestone:	1.9 M3
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	Regression (Used to work)
Verified:		Documentation change:	Don't know

Description Enrico Scholz 2015-04-09 12:15:24 UTC

After commit 7c0fd561bad0250a00cef63e3d787573112a59cf hardlinks are not preserved anymore and files are packaged multiple times.  ELF stripping happens only for one instance and remaining ones will be packaged with their debug info.

See e.g. e2fsprogs-e2fsck which ships fsck.ext[234] as a hardlinks to the common e2fsck tool.

Comment 1 Martin Jansa 2015-04-10 14:09:43 UTC

I agree this should be fixed in package.py, but for e2fsprogs there is simple work around at least temporary:
http://lists.openembedded.org/pipermail/openembedded-core/2015-April/103595.html

Comment 2 Mark Hatle 2015-04-10 16:22:10 UTC

The right fix to this is likely to determine a mechanisms to get a deterministic output.  Something like shortest pathname, alphabetic, etc..

And then use that to sort the hard links so that the strip/split/gnu debuglink get a deterministic result.

Comment 3 Ed Bartosh 2015-04-15 15:12:40 UTC

Looking at the output of "find ./tmp -type f -links +1 -printf '%i %n %p\n' |sort" I'd say it doesn't make much sense to fix this issue as after Martin's fix there no files with more than 2 hardlinks.

Comment 4 Enrico Scholz 2015-04-15 15:39:06 UTC

I get a lot of hardlinks after removal of 7c0fd5...

10486487 5 ./tmp/sysroots/phytec-phyflex-mx6/usr/share/terminfo/m/mime
10486487 5 ./tmp/sysroots/phytec-phyflex-mx6/usr/share/terminfo/m/mimei
10486569 4 ./tmp/sysroots/phytec-phyflex-mx6/usr/share/terminfo/m/minitel-2


I do not think that the changes to e2fsprogs should persist when this issue has been fixed. Symlinks are more expensive than hardlinks.

Comment 5 Ed Bartosh 2015-04-15 21:40:59 UTC

Thanks for pointing that out. I missed that. Will investigate further.

Comment 6 Ed Bartosh 2015-04-20 12:35:30 UTC

Here is a diff between 2 lists <path> <number of hardlinks> made before and after applying 7c0fd561bad0250 for core-image-sato:

-./tmp/work/i586-poky-linux/binutils/2.25-r0/packages-split/binutils/usr/bin/i586-poky-linux-ld 4
-./tmp/work/i586-poky-linux/binutils/2.25-r0/packages-split/binutils/usr/bin/i586-poky-linux-ld.bfd 4
+./tmp/work/i586-poky-linux/binutils/2.25-r0/packages-split/binutils/usr/bin/i586-poky-linux-ld 2
+./tmp/work/i586-poky-linux/binutils/2.25-r0/packages-split/binutils/usr/bin/i586-poky-linux-ld.bfd 2
-./tmp/work/i586-poky-linux/binutils/2.25-r0/package/usr/bin/i586-poky-linux-ld 4
-./tmp/work/i586-poky-linux/binutils/2.25-r0/package/usr/bin/i586-poky-linux-ld.bfd 4
+./tmp/work/i586-poky-linux/binutils/2.25-r0/package/usr/bin/i586-poky-linux-ld 2
+./tmp/work/i586-poky-linux/binutils/2.25-r0/package/usr/bin/i586-poky-linux-ld.bfd 2
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/e2fsck 10
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/e2fsck 8
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/e2label 4
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/e2label 2
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext2 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext3 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext4 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext4dev 10
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext2 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext3 2
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext4 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext4dev 8
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mke2fs 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext2 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext3 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext4 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext4dev 10
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mke2fs 2
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext2 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext3 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext4 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/mkfs.ext4dev 8
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/tune2fs 4
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/tune2fs 2
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/e2fsck 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext2 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext3 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext4 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext4dev 10
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/e2fsck 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext2 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext3 2
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext4 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext4dev 8
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mke2fs 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext2 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext3 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext4 10
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext4dev 10
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mke2fs 2
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext2 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext3 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext4 8
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-mke2fs/sbin/mkfs.ext4dev 8
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-tune2fs/sbin/e2label 4
-./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-tune2fs/sbin/tune2fs 4
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-tune2fs/sbin/e2label 2
+./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-tune2fs/sbin/tune2fs 2
-./tmp/work/i586-poky-linux/gawk/4.1.1-r0/packages-split/gawk/usr/bin/gawk 4
-./tmp/work/i586-poky-linux/gawk/4.1.1-r0/packages-split/gawk/usr/bin/gawk-4.1.1 4
+./tmp/work/i586-poky-linux/gawk/4.1.1-r0/packages-split/gawk/usr/bin/gawk 2
+./tmp/work/i586-poky-linux/gawk/4.1.1-r0/packages-split/gawk/usr/bin/gawk-4.1.1 2
-./tmp/work/i586-poky-linux/gawk/4.1.1-r0/package/usr/bin/gawk 4
-./tmp/work/i586-poky-linux/gawk/4.1.1-r0/package/usr/bin/gawk-4.1.1 4
+./tmp/work/i586-poky-linux/gawk/4.1.1-r0/package/usr/bin/gawk 2
+./tmp/work/i586-poky-linux/gawk/4.1.1-r0/package/usr/bin/gawk-4.1.1 2
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/i915_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/i965_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/nouveau_vieux_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/r200_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/radeon_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/swrast_dri.so 12
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/i915_dri.so 2
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/i965_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/nouveau_vieux_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/r200_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/radeon_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/packages-split/mesa-megadriver/usr/lib/dri/swrast_dri.so 10
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/i915_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/i965_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/nouveau_vieux_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/r200_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/radeon_dri.so 12
-./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/swrast_dri.so 12
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/i915_dri.so 2
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/i965_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/nouveau_vieux_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/r200_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/radeon_dri.so 10
+./tmp/work/i586-poky-linux/mesa/2_10.4.4-r0/package/usr/lib/dri/swrast_dri.so 10
-./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/i915_dri.so 6
-./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/i965_dri.so 6
-./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/nouveau_vieux_dri.so 6
-./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/r200_dri.so 6
-./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/radeon_dri.so 6
-./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/swrast_dri.so 6
+./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/i965_dri.so 5
+./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/nouveau_vieux_dri.so 5
+./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/r200_dri.so 5
+./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/radeon_dri.so 5
+./tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/rootfs/usr/lib/dri/swrast_dri.so 5

not that many files affected, I'd say

Comment 7 Ed Bartosh 2015-04-22 12:15:16 UTC

First attempt to fix this: http://lists.openembedded.org/pipermail/openembedded-core/2015-April/104017.html

Here is how harlinks look with the fix:
$ find ./tmp -name *fsck* -type f -executable -links +1 -printf '%i %n %k %p\n' |sort
2610102 2 80 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/package/sbin/fsck.minix.util-linux
2610102 2 80 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/packages-split/util-linux-fsck/sbin/fsck.minix.util-linux
2610131 2 128 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/package/sbin/.debug/fsck.minix.util-linux
2610131 2 128 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/packages-split/util-linux-dbg/sbin/.debug/fsck.minix.util-linux
2610222 2 28 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/packages-split/util-linux-fsck.cramfs/usr/sbin/fsck.cramfs
2610222 2 28 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/package/usr/sbin/fsck.cramfs
2610506 2 32 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/package/sbin/fsck.util-linux
2610506 2 32 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/packages-split/util-linux-fsck/sbin/fsck.util-linux
2610508 2 52 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/package/sbin/.debug/fsck.util-linux
2610508 2 52 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/packages-split/util-linux-dbg/sbin/.debug/fsck.util-linux
2610515 2 64 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/packages-split/util-linux-dbg/usr/sbin/.debug/fsck.cramfs
2610515 2 64 ./tmp/work/i586-poky-linux/util-linux/2.26.1-r0/package/usr/sbin/.debug/fsck.cramfs
2636691 5 812 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/image/sbin/e2fsck
2636691 5 812 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/image/sbin/fsck.ext2
2636691 5 812 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/image/sbin/fsck.ext3
2636691 5 812 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/image/sbin/fsck.ext4
2636691 5 812 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/image/sbin/fsck.ext4dev
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/e2fsck
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext2
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext3
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext4
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/fsck.ext4dev
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/e2fsck
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext2
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext3
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext4
2639022 10 272 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-e2fsck/sbin/fsck.ext4dev
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/.debug/e2fsck
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/.debug/fsck.ext2
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/.debug/fsck.ext3
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/.debug/fsck.ext4
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/package/sbin/.debug/fsck.ext4dev
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-dbg/sbin/.debug/e2fsck
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-dbg/sbin/.debug/fsck.ext2
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-dbg/sbin/.debug/fsck.ext3
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-dbg/sbin/.debug/fsck.ext4
2640003 10 544 ./tmp/work/i586-poky-linux/e2fsprogs/1.42.9-r0/packages-split/e2fsprogs-dbg/sbin/.debug/fsck.ext4dev

Please, review.

Comment 8 Ed Bartosh 2015-04-22 19:58:46 UTC

A bit shorter variant without calling a shell as was suggested by reviewer: http://lists.openembedded.org/pipermail/openembedded-core/2015-April/104036.html

Comment 9 Ed Bartosh 2015-04-29 15:16:31 UTC

Should be fixed by this commit: http://cgit.openembedded.org/openembedded-core/commit/?id=82d00f7254b7d3bb6a167d675d798134884d1b19

Comment 10 Paul Eggleton 2015-05-18 16:21:51 UTC

This has also been backported to dizzy:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dizzy&id=9c4ff467f66428488b1cd9798066a8cb5d6b4c3b

Comment 11 Costin 2015-10-26 14:49:21 UTC

Hello,

Verified with 505a82673ac2487df5ea343a6422c2fc47018831 on an Ubuntu machine

Steps done, reflect comment#7 approach:

1) build an image, i chose core-image-minimal
source oe-init-build-env
bitbake core-image-minimal

2)find ./tmp -name *fsck* -type f -executable -links +1 -printf '%i %n %k %p\n' |sort

5898756 10 240 ./tmp/sysroots/x86_64-linux/sbin/e2fsck
5898756 10 240 ./tmp/sysroots/x86_64-linux/sbin/fsck.ext2
5898756 10 240 ./tmp/sysroots/x86_64-linux/sbin/fsck.ext3
5898756 10 240 ./tmp/sysroots/x86_64-linux/sbin/fsck.ext4
5898756 10 240 ./tmp/sysroots/x86_64-linux/sbin/fsck.ext4dev
...