4628 2013-06-07 20:42:29 +0000 non-deterministic shlibs providers 2014-07-25 15:28:36 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 1.4 x86 Multiple RESOLVED FIXED Medium normal 1.7 1 Martin.Jansa richard.purdie meta.mr.watcher meta.watcher richard.purdie --- --- Yes (doc changes required) oldest_to_newest 33715 0 Martin.Jansa 2013-06-07 20:42:29 +0000 Reported on ML: http://lists.openembedded.org/pipermail/openembedded-core/2013-June/079660.html Imagine this process foo.bb DEPENDS on libabc.bb to provide libabc.1.so evil recipe bar.bb installs some binary crap in /opt/crap and because it's picky about libabc version it bundles own compy of libabc.so.1 and installs it to /opt/crap/lib/libabc.so.1 At the time of first build nobody notices libabc.1.so and foo, bar and libabc are happily populated to sysroot. foo.bb is rebuilt because of some unrelated change, but this times it uses bar as shlibs provider for libabc.so.1 foo does not work in runtime, because it cannot find libabc.so.1 hidden in /opt/crap/lib. bar.bb is "fixed" by adding EXCLUDE_FROM_SHLIBS to prevent further polluting of shlibs providers, but damage is already done. foo doesn't have any dependency on bar (DEPENDS/RDEPENDS) because bar.bb is just unrelated binary crap which just happens to bundle libabc.so.1 too (so foo checksum does not include bar checksum) Now the tricky part: 1) fixing local build is easy bitbake -c cleansstate `grep ' bar' buildhistory/images/machine/eglibc/image-with-foo/depends.dot | sed 's/ -> .*//g' | xargs` but sstate checksums are identical with bar or libabc in package "Depends:" field, so how to cleanup SSTATE_MIRROR and sstate archives already distributed to every local builder? I could bump PR in libabc, but with PR bumps and PRINC going away I really need to know how to solve such issues in future. Should we fix do_package to filter shlibs providers to include only recipes which are in (R)DEPENDS/RRECOMMENDS/RSUGGESTS? Phil Blundell: > It sounds like the real bug here is that the shlibdeps code doesn't > issue a diagnostic when it encounters this kind of ambiguous situation. > It ought to be perfectly capable of figuring out that there are two > copies of libabc.so.1 at large which belong to different packages and, > absent additional guidance, it has no reliable way of knowing which one > of those is going to satisfy the DT_NEEDED in any random binary. > > If shlibdeps stopped with an error in that situation, which arguably it > ought to, then the erroneous dependency would never get into sstate and > the scenario that you describe would not be able to happen. Richard Purdie: > We should perhaps limit shlibs to the default linker search paths? > > Wouldn't there be a warning if two things provide the same shlibs or > does that assume the shlibs versioning sorts itself out? I like both suggestions from Richard and from Richard, there is another slightly related issue with https://bugzilla.yoctoproject.org/show_bug.cgi?id=4102 It's probably possible to create test case where I create sstate package with libabc.so.1 provided by MACHINE_ARCH foo and then try to reuse this sstate but with libabc.so.1 provided by TUNE_PKGARCH foo (e.g. with different version) shlibs code will still find it provider by MACHINE_ARCH foo and there isn't way to clean foo from incremental build (only manually). 1) RP1: We should perhaps limit shlibs to the default linker search paths? 2) RP2+PB: Warning/Error if two things provide the same shlibs - probably better to error out when bar.bb is being built and tries to provide the same libabc libabc.bb, it can be in different order as bar does not depends on libabc, so next time it can show error when building libabc, but IMHO better then showing error when foo finds out 2 possible libabc providers 3) Restrict possible providers to stuff included in dependencies - I don't know how difficult it will be to implement this (to see all dependencies including transitive), but IMHO this one is most important and can also solve #4102, most tricky part would be to prevent shlib providers silently disappearing from RDEPENDS - when something doesn't declare dependency explicitly, but it was always there when do_package was executed - now missing providers are silently ignored IIRC. Buildhistory can probably help a lot to find missing explicit deps for shlibs providers when we restrict it. This will also ensure that when shlibs provider is changed, the package which is automagically RDEPENDing on it gets rebuilt too. I still have test-case for this, so I can look into this in following days. 34370 1 Martin.Jansa 2013-07-06 18:59:34 +0000 I've added some changes to package.bbclass to http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/shlib-providers it should implement: 1) RP1: We should perhaps limit shlibs to the default linker search paths? 2) RP2+PB: Warning/Error if two things provide the same shlibs and there is some pseudo code to describe the idea what I would like to do to implement: 3) Restrict possible providers to stuff included in dependencies 36394 2 richard.purdie 2013-09-07 08:51:48 +0000 Martin and I discussed this further on the mailing list. The principle is good but we need to work on the patches and its 1.6 material at this point. 39917 3 Martin.Jansa 2014-01-18 14:06:25 +0000 I've sent latest version of my patches to ML http://lists.openembedded.org/pipermail/openembedded-core/2014-January/088522.html it doesn't resolve all issues for this task, but at least those most important for us (webos). It's tested in meta-webos (backported to dylan), meta-webos-ports (backported to dora and master), my world builds from master. 44712 4 richard.purdie 2014-07-25 15:28:36 +0000 Firstly, we've added rpath support to shlibs: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3ead62a57833e2a3fd25e592a754c3ecfcd77b3a which means that different libraries can coexist and only conflict if they really conflict. Secondly, we now have a new high level sanity check that checks if dependencies are all listed in build dependencies: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a202f8a66383787c899246c5fa14c862e5d7647d I believe that between the two things, this issue is now resolved.