Bug 1250

Summary: DEPENDS not honored when rebuilding from sstate with SSTATEPOSTINSTFUNCS
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Scott Garman <scott.a.garman>
Component: meta-yoctoAssignee: Saul Wold <sgw>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: High CC: poky.bs.watcher, poky.watcher, sgw
Version: 1.0   
Target Milestone: 1.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Scott Garman 2011-07-15 16:23:01 UTC 
The following scenario was identified on the autobuilder during this build:

http://autobuilder.pokylinux.org:8010/builders/crownbay/builds/25/steps/shell_17/logs/stdio

At line 1711, docbook-sgml-dtd-3.1-native.bb failed because install-catalog did not exist. install-catalog is provided by sgml-common-native in the native sysroot, and the docbook-sgml-dtd-native.inc recipe has a DEPENDS = "sgml-common-native" in it.

However, if you look at that log, you'll find that sgml-common-native's do_populate_sysroot is not run until line 1755.

This seems to indicate that the DEPENDS is not being properly honored.

Of note is that this build is being done from an empty TMPDIR but a fully populated sstate.
Comment 1 Saul Wold 2011-07-16 17:35:33 UTC 
I was able to reproduce this even after adding a direct depends on populate_sysroot

do_unpack[depends] += "sgml-common-native:do_populate_sysroot"

Was added to openjade, docbook-sgml-dtd, and docbook-dsssl-sytlesheets.

I then did a cleansstate on the following recipes:

bitbake openjade-native sgml-common-native docbook-dsssl-stylesheets-native docbook-sgml-dtd-4.5-native docbook-utils-native -c cleanstate

Followed by a bitbake of docbook-utils-native (has dependencies on the other docbook items).

Then I did a bitbake -c clean of the above list and tried to build docbook-utils-native again, and I could reproduce the failure right away.


 bitbake docbook-utils-native
Loading cache: 100% |####################################################################################################| ETA:  00:00:00
Loaded 1028 entries from dependency cache.

OE Build Configuration:
BB_VERSION        = "1.13.2"
TARGET_ARCH       = "x86_64"
TARGET_OS         = "linux"
MACHINE           = "sugarbay"
DISTRO            = "poky"
DISTRO_VERSION    = "1.0+snapshot-20110717"
TARGET_FPU        = ""
meta              
meta-yocto        = "rc3:aca9d745202ebdc6d3638a62c22af7ab8fb9b1a1"
meta-sugarbay     = "master:92fc07a5f1b98779806cdcc2341487ff5ea5a238"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 14 of 19 (/intel/poky/distro/meta/recipes-devtools/docbook-utils/docbook-utils-native_0.6.14.bb:do_populate_sysroot_setscene)
NOTE: Running setscene task 15 of 19 (/intel/poky/distro/meta/recipes-devtools/docbook-utils/docbook-utils-native_0.6.14.bb:do_populate_lic_setscene)
NOTE: Running setscene task 16 of 19 (/intel/poky/distro/meta/recipes-devtools/openjade/openjade-native_1.3.2.bb:do_populate_sysroot_setscene)
NOTE: Running setscene task 17 of 19 (/intel/poky/distro/meta/recipes-devtools/docbook-dsssl-stylesheets/docbook-dsssl-stylesheets-native_1.79.bb:do_populate_sysroot_setscene)
NOTE: package openjade-native-1.3.2-r3: task do_populate_sysroot_setscene: Started
NOTE: package docbook-dsssl-stylesheets-native-1.79-r3: task do_populate_sysroot_setscene: Started
NOTE: package docbook-utils-native-0.6.14-r1: task do_populate_sysroot_setscene: Started
NOTE: package docbook-utils-native-0.6.14-r1: task do_populate_lic_setscene: Started
NOTE: package docbook-utils-native-0.6.14-r1: task do_populate_sysroot_setscene: Succeeded
NOTE: package docbook-utils-native-0.6.14-r1: task do_populate_lic_setscene: Succeeded
ERROR: Task 65 (/intel/poky/distro/meta/recipes-devtools/docbook-dsssl-stylesheets/docbook-dsssl-stylesheets-native_1.79.bb, do_populate_sysroot_setscene) failed with exit code '1'
ERROR: Task 51 (/intel/poky/distro/meta/recipes-devtools/openjade/openjade-native_1.3.2.bb, do_populate_sysroot_setscene) failed with exit code '1'
NOTE: Running setscene task 19 of 19 (/intel/poky/distro/meta/recipes-devtools/sgml-common/sgml-common-native_0.6.3.bb:do_populate_sysroot_setscene)
NOTE: package sgml-common-native-0.6.3-r0: task do_populate_sysroot_setscene: Started
NOTE: package sgml-common-native-0.6.3-r0: task do_populate_sysroot_setscene: Succeeded
NOTE: Executing RunQueue Tasks
NOTE: Running noexec task 110 of 111 (ID: 8, /intel/poky/distro/meta/recipes-devtools/docbook-utils/docbook-utils-native_0.6.14.bb, do_package_write)
NOTE: Running noexec task 111 of 111 (ID: 5, /intel/poky/distro/meta/recipes-devtools/docbook-utils/docbook-utils-native_0.6.14.bb, do_build)
NOTE: Tasks Summary: Attempted 111 tasks of which 109 didn't need to be rerun and 0 failed.
Comment 2 Richard Purdie 2011-07-18 06:15:58 UTC 
By definition providing a sstate package means you that DEPENDS need not be honored. This isn't therefore a bug in the dependency handling but in the way sstate is being used :/
Comment 3 Richard Purdie 2011-07-18 06:26:35 UTC 
Another worrying factor is that even if the sstate package fails, it should just get rebuilt and things should continue so this error should not be fatal...
Comment 4 Richard Purdie 2011-07-18 07:00:44 UTC 
Ok, I propose that we:

a) Have any of the recipes with these postinstalls take a copy of install-catalog out of the sysroot and place it into recipe's sysroot contents with a suitable suffix, something like:

SYSROOT_PREPROCESS_FUNCS += "xxx_sysroot_preprocess"

xxx_sysroot_preprocess () {
	install -d ${SYSROOT_DESTDIR}${bindir_crossscripts}/
	install -m 755 ${STAGING_BINDIR_NATIVE}/install-catalog ${SYSROOT_DESTDIR}${bindir_crossscripts}/install-catalog-xxx
}

which will always ensure this is present when the sstate package is processed.

This could perhaps now best be done as a bbclass.

b) We need to figure out why when the sstate package failed to install the system didn't recover as it should have done...
Comment 5 Richard Purdie 2011-07-18 08:02:31 UTC 
On the failed sstate package handling issue, there is a fundamental problem with some of the assumptions sstate operates under.

For target packages we'd never run them so its safe to assume that any (build) DEPENDS are now unneeded and also runtime RDEPENDS and so forth are not required. Its these cases the design of sstate is catering well for.

The problem comes with native/cross sstate packages where build time dependencies shouldn't be required but runtime dependencies can be. Worse still, looking at RDEPENDS isn't enough since we autodetect shared library dependencies so the RDEPENDS information in the metadata isn't complete.

The easiest solution is likely to make sstate start looking at DEPENDS+RDEPENDS in the native/cross case.

Its worth nothing that the checksum code gives a significant amount of protection assuming sstate data is mirrored completely as only in the case of a failed package installation would an issue arise.