Bug 6465

Summary: Enabling multilib does not taint all sstate content
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Paul Gortmaker <paulg>
Component: coreAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: meta.mr.watcher, meta.watcher
Version: unspecified   
Target Milestone: 1.8   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Paul Gortmaker 2014-06-18 19:46:39 UTC 
Enabling multlib in conf/local.conf with:

 require conf/multilib.conf
 MULTILIBS = "multilib:lib32"
 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

as per the docs https://wiki.yoctoproject.org/wiki/Multilib

needs to taint all shared library binaries living in sstate.  Otherwise you get hard to diagnose behaviour where those binaries won't execute or even be probable with "ldd".

Here is an example, using the stat binary from the "stat" package:

-------------------------
bash-4.3# cd /bin
bash-4.3# ./stat.stat
bash: ./stat.stat: No such file or directory
bash-4.3# ls -l ./stat.stat
-rwxr-xr-x 1 root root 16712 Jun 17 14:30 ./stat.stat
bash-4.3# ldd ./stat.stat
/usr/bin/ldd: line 117: ./stat.stat: No such file or directory
bash-4.3# ln -s /lib64/ld-linux-x86-64.so.2  /lib/ld-linux-x86-64.so.2  #  HACK!
bash-4.3# ./stat.stat
Usage: ./stat.stat [-l] [-f] [-s] [-v] [-h] [-t] [-c format] file1 [file2 ...]
bash-4.3# ldd ./stat.stat
	linux-vdso.so.1 (0x00007fff14361000)
	libc.so.6 => /lib64/libc.so.6 (0x000000370f600000)
	/lib/ld-linux-x86-64.so.2 (0x000000370f200000)
bash-4.3# 
---------------------

compare the last line from ldd to that from a recompiled stat from a fresh build with all sstate purged:

--------------------
bash-4.3# ldd ./stat.stat-recompiled 
	linux-vdso.so.1 (0x00007fff6a5fe000)
	libc.so.6 => /lib64/libc.so.6 (0x000000370f600000)
	/lib64/ld-linux-x86-64.so.2 (0x000000370f200000)
bash-4.3# 
--------------------

Note the last line in each ldd output and the lib vs lib64 prefix.

Similarly, if anyone goes from multilib enabled to multilib disabled, all binaries using dynamic library need their sstate tainted.
Comment 1 Richard Purdie 2014-09-26 10:14:28 UTC 
I've looked into this and I can't see anything wrong. I started with a qemux86-64 build with no multilib enabled. This installed libraries into /lib/ as expected.

I then added the lines you mentioned. This causes 64 bit to be installed into lib64 as expected. When I compared the signatures for the stat task I see:

$ bitbake-diffsigs tmp/stamps/core2-64-poky-linux/stat/3.3-r0.do_populate_sysroot.sigdata.2* tmp/stamps/core2-64-poky-linux/stat/3.3-r0.do_populate_sysroot.sigdata.b*
basehash changed from 113c64dfdd151517f1863005203894a5 to 2cf30b9c6750056d063202a5c728857d
Variable baselib value changed from 'lib' to 'lib64'

i.e. the hash did change and it did so since baselib changed. This would trigger a rebuild (the 2* and b* hashes were before and after the change).

I did find an unrelated problem where another change elsewhere triggered a rebuild of everything and will send a separate patch for that.

So as far as I can tell, sstate is tainted for multilib?
Comment 2 Richard Purdie 2014-09-26 11:17:01 UTC 
On a hunch, I tried looking at the actual build output and you are correct, it does not update correctly after the multilib changes. The reason is quite simple too, stat doesn't have a separate source and build directory so the system just goes into the directory and does "make". The core system libs and loader shims have changed but make doesn't realise than and the binary isn't rebuilt.

So sstate is perfectly fine, the reuse of the existing build directory is problematic though and its effectively a make issue.

Starting from a clean tmpdir obviously avoids the problem. Using something which has separate build support like autotools (B != S) also avoids the problem.
Comment 3 Richard Purdie 2014-11-07 14:54:13 UTC 
The following patches have been merged to various key classes to try and improve this:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=444ae98131f7f9e962d102f408eb867ab53a452a
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6bd2d9d395fe7c97a0a9645ca9379b366bb64afa
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c1f4b1f76627c76f23c2037a1e9dad10a31ec258
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d5881b7b94b195ad3858d46992ba6edbb2c95f31

Its probably not perfect but it is probably as good as we can get without complete removal of the source and the unpack/patch again.