For people who are trying to muck about in the depths of the metadata, how the bitbake tasks progress and what directories they touch/require can be fairly opaque. For instance, it was unclear to me that if I was setscening, the do_package task was skipped. The sections 22 of the megamnual doe a nice job of detailing the flow for a "standard" build. It would be helpful to add the setscening versions as a parallel, especially if SSTATE_MIRRORS become more standard.
So to be clear, this is talking about adding details of sstate to http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#bitbake-dev-environment, section 3.5 "Bitbake". The comment from Brian refers to version 2.0 of the megamanual. We'd probably want to add a section, "3.5.6 Setscene tasks and SState". The text could read: """ The description of tasks so far assumes that bitbake needs to build everything and there are no prebuilt objects available. Bitbake does support skipping tasks if prebuilt objects are available. These are usually make available in the form of an sstate cache (see the SSTATE_DIR and SSTATE_MIRRORS variables). The idea of setscene task is a version of the task where instead of building something, we can skip to the end result and simply place a set of files into some specific locations. In some cases it makes sense to have a setscene task variant (e.g. generating package files in do_package_write_*), in some cases it doesn't make sense (e.g. a do_patch or do_unpack task) since the work involved would be equal to or greater than the underlying task. In OpenEmbedded, the common tasks which have setscene variants are do_package, do_package_write_*, do_deploy, do_packagedata and do_populate_sysroot, most of the end result output tasks. The system has knowledge of the relationship between these tasks and others which precede them. For example, if bitbake runs do_populate_sysroot_setscene for something, there is little point in running any of do_fetch/do_unpack/do_patch/do_configure/do_compile/do_install. If however do_package needed to run, it would need to run those other tasks. It becomes more complicated since for example if everything can come from an sstate cache, some things are simply not required at all. You don't need a compiler for example, or native tools such as quilt if there is nothing to compile or patch. If the do_package_write_* packages are available from sstate, you don't need the do_package task data. To handle all these complexities, bitbake runs in two phases. The first is the "setscene" stage. Here, firstly bitbake checks the sstate cache for any targets its planning to build. It will do a fast "does it exist?" check initially rather than a complete download. If nothing exists, the setscene stage would complete and the main build would proceed. If items are found in the sstate cache, the system works backwards from the end targets specified by the user. If for example an image is being built, the system would first look for the packages needed for that image and the tools needed to construct an image. If those were available, the compiler wouldn't be needed so that wouldn't even get downloaded. If something was found to be unavailable, or the download or setscene task failed, it would then try to install dependencies like the compiler from the cache. The availability of objects in the sstate cache is handled by the function specified by the BB_HASHCHECK_FUNCTION variable and returns a list of the objects that are available. The function specified by the BB_SETSCENE_DEPVALID variable is the one which determines whether a given dependency needs to be followed or not and for any given relationship passed to it, returns a True or False value. Once the setscene process completes, the system has a list of tasks which it believes it can "accelerate" and that it therefore doesn't need to run. There is a final function call to the function specified by the BB_SETSCENE_VERIFY_FUNCTION2 variable which is able to require that tasks which it was going to skip do actually get run. """ Scott: Does the above give you enough info to add something to the manual? Brian: Does this give you the information you're asking for? I appreciate its not quite as specific as you'd like but its actually a rather complex topic.
That actually helps alot. The last 2 paragreaphs probably need a little additional info, if possible. For example, BB_HASHCHECK_FUNCTION is pretty clear. Conceptually, the BB_SETSCENE_DEPVALID makes sense but a short example of a task with a dependency dictionary might make it easier to understand what it's doing. I'm afraid I'm not clear on exactly what BB_SETSCENE_VERIFY_FUNCTION2 is doing or why there is also a BB_SETSCENE_VERIFY_FUNCTION ?
Richard and Brian, See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#setscene-tasks-and-shared-state for the new section. Section numbering was off a bit in the comments so I added it as a subsection at the very end of the "BitBake" section as 3.5.7. I basically took your write-up Richard and applied my edits. One thing I am not sure of is whether to consistently use "OpenEmbedded build system" or "BitBake" when talking about the system doing stuff during the build. Right now it is a bit of a mix. Let me know and I will use one term. Or, perhaps, both are needed -- I don't know. Regarding the last paragraph, I could not tell exactly what was doing what and what was needing passed, etc. So, that might be technically off as I have written it. Also, I assumed that the BB_SETSCENE_VERIFY_FUNCTION2 variable mentioned in the comment really meant BB_SETSCENE_VERIFY_FUNCTION, which we have documented in the BitBake User Manual. I also do not have a short example showing BB_SETSCENE_DEPVALID as Brian requested. Let me know of any changes. Thanks, Scott
Merged in yocto-docs 6aa0e6b837de14dd7d6c5ef34328bbf268dd6d71.
Setting the doc flag to "done". I am assuming my questions in 3 did not need answered since the bug was set to RESOLVED. Scott