We make relatively few references to how staging works in our manuals; pretty much the only coverage is brief reference items on do_populate_sysroot and staging.bbclass. What I think we need to do: 1) Cover which directories get staged - under do_populate_sysroot we say it's a subset, but we don't specify what that subset is (sysroot_stage_dirs() in meta/classes/staging.bbclass is what controls that, we probably need to explain it since the directories staged depend on what type of recipe it is). We should explain why we only stage a subset and how to stage additional directories in the rare cases when that's needed for a recipe. 2) Mention that you must not copy files directly into the sysroot (e.g. directories pointed to by STAGING_* variables) as this bypasses shared state and prevents the build system from being able to un-stage those files when needed. 3) Consider enhancing the "Closer Look..." section to mention staging, although unfortunately the diagram does not actually include staging as a step.
Richard points out that "staging" is a bit of an archaic term referring back to how things were done in OE-Classic, we should probably avoid it.
Hi Paul, What is a better term now rather than "staging?" The directories are being prepared for something r right? We have lots of variables that have "STAGING" in their name. I looked at sysroot_stage_dirs in the staging.bbclass file and it looks like the variable STAGING_DIRS plays a role. We don't have that documented in the glossary right now. We do have a STAGING_DIR. Is that variable incorrectly named? Is the STAGING_DIRS variable what determines which directories are being staged? I would think that what we enhance the "Closer Look..." section to include regarding staging could be the main area to deal with this missing documentation. Scott
Setting to Need information.
SYSROOT_DIRS, SYSROOT_DIRS_NATIVE and SYSROOT_DIRS_BLACKLIST are fairly new variables, and yes these control the list of directories - previously that was hardcoded. When we're documenting how to extend the list of directories that get populated, we should recommend using these, and yes we want them in the glossary. See here for the commit that added them: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e8394314684f4c9855b9420d913cb45dad8bebf6 Instead of "staging" we should talk about "sysroot population", but in the text we should mention that "staging" was the earlier term for this and it can still be found in various places. The "closer look" definitely needs to make reference to these, however we also need to cover it elsewhere, in particular the "Writing a new recipe" section - for that however you could pretty much copy and paste the "Sharing Files Between Recipes" subsection from the SDK manual since that's about the right level of detail for that section.
Hi, This is a fairly involved documentation bug. I am going to indicate what has been done here so far: * Created a new section in the "Writing a New Recipe" section of the dev-manual. This new section is called "Sharing Files Between Recipes." It is here - http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#new-sharing-files-between-recipes. I based this section on the same section from the sdk-manual. I changes some of the wording. Note that I gave a bit of history for the term "staging" in the section and pointed to three new variables that are now in the glossary: SYSROOT_DIRS, SYSROOT_DIRS_NATIVE, and SYSROOT_DIRS_BLACKLIST. * I created a new variable glossary entry for the each of the SYSROOT_DIRS* variables: See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-SYSROOT_DIRS, http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-SYSROOT_DIRS_NATIVE, and http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-SYSROOT_DIRS_BLACKLIST. Note that these definitions are very terse and based only on what I could get out the commit in Comment 4. Each definition will likely need some more detail. * What I am missing is the updates to the "Closer Look" section of the ref-manual. Paul suggested a "cheap" fix by getting some do_populate_sysroot stuff into the section "Configuration and Compilation" at http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#configuration-and-compilation-dev-environment. Please comment on what has been done here so far and any further explorations for updating "Closer Look". thanks, Scott
Moving to 2.3 M2
Need some comments on Comment 5. Moved to 2.3 M3 Scott
Moved to 2.3 M4
This is quite involved so I've sent Scott email, particularly given the recipe specific sysroot changes.
Hi, I have made adjustments to the bug. I will duplicate the dialog here for any copied on the bug. It is now considered IN PROGRESS REVIEW. ==================================== I have these sections updated. Here are the relevant links so you and/or Paul can review them and provide me with any changes: * The section "Configuration and Compilation" (http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#configuration-and-compilation-dev-environment). This documents the new "do_prepare_recipe_sysroot" task. What about the build directory in that figure right above the text? Wouldn't that be changing also? I am also wondering about the "Package Splitting" section that follows (http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#package-splitting-dev-environment). Is that section still accurate? * The new structure section is here - http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#structure-build-tmp-work-tunearch-recipename-version. This lays out the recipe work directory with all its subdirectories. * "staging.bbclass" changes are here - http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ref-classes-staging. * The updates for "Sharing Files Between Recipes" are here - http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#new-sharing-files-between-recipes. Here are some additional items: * The terms "recipe-sysroot" and "recipe-sysroot-native" appear in a few places now. Are these actual directory names? Or, are they simply terms you use to describe the two types of sysroots? The reason I ask is because I am formatting them as a directory. * We don't have a task description for "do_prepare_recipe_sysroot" task. I created a brief item in the ref-manual here - http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ref-tasks-prepare_recipe_sysroot * We don't document the "SYSROOT_DESTDIR" variable. Outside of saying "Defines the destination directory of the sysroot" what would we want there? * We mention "tmp/sysroot-components" as a directory used. It is not in the section of the ref-manual that talks about all the directories and what they are used for. * We don't document the "SSTATE_SCAN_FILES" variable. I am not sure if we want to... but, I put in this entry. Let me know. * The first paragraph for "Stage Two" in http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ref-classes-staging throws some formated terms around and I don't know if they should be formatted or not. They are "recipe-sysroot", "recipe-sysroot-native", and "sysroot-components". I mentioned these a previously a bit in the email. * In the first paragraph of "Sharing Files Between Recipes" (http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#new-sharing-files-between-recipes), again I injected the terms "recipe-sysroot" and "recipe-sysroot-native" to try and add clarity. I am using these right? * Finally, you might want to take a look at the migration section for 2.3 that talks about sysroot - http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#migration-2.3-sysroots. Thanks, Scott
Since the tarball for 2.3 has been made for RC, I am moving the target date out to 2.4 for this bug. I would like to get the changes reviewed before I mark it RESOLVED. Scott
Needs a review please. Scott
My final link in Comment 10 is not right - use http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#migration-2.3-recipe-specific-sysroots. This stuff needs to be looked at. Scott
Looking at each item in your most recent summary (let me know if I missed anything): == Sharing files between recipes == * In several places you have "populating sysroot" when presumably you mean either "populating a sysroot" or "populating sysroots". * The final paragraph should presumably begin with a capital letter * To the end of "The following example shows how you could add the /opt directory to the list of directories" add " within a recipe" just so it's clear where the line would be added. == Package Splitting == This section is still accurate - it was unaffected by the RSS changes. == build/tmp/work/tunearch/recipename/version/ == I think it would be better to be using ${WORKDIR} rather than "recipe_work_directory", since that's actually what ${WORKDIR} will expand to in the recipe context. Otherwise this section looks fine. == staging.bbclass == * "recipe work directory (i.e. WORKDIR" - missing closing parenthesis * "Although these files are not recommended" -> "Although such sysroot postinst- scripts are not recommended" == Migration section == * "Open-Embedded" -> "OpenEmbedded" * "Specify Which STAGING_DIR_NATIVE is Used" -> "Specify the sysroot when using certain external scripts" == Misc == * "recipe-sysroot" and "recipe-sysroot-native" are indeed actual subdirectories of the work directory as you've already surmised. * For do_prepare_recipe_sysroot I would say "Installs the files into the individual recipe specific sysroots (i.e. recipe-sysroot and recipe-sysroot-native under ${WORKDIR}) based upon the dependencies specified by DEPENDS." * Here's a description for SYSROOT_DESTDIR: "Points to the temporary directory under the work directory (default ${WORKDIR}/sysroot-destdir) where files to be populated into the sysroot are assembled during do_populate_sysroot." * And for tmp/sysroots-components (note the slightly counter-intuitive plural, you'll need to correct your two current references): "Location of the sysroot contents that do_prepare_recipe_sysroot will hard link or copy into the recipe specific sysroot for each recipe in DEPENDS. Population of this directory is handled via shared state and the path is specified by the COMPONENTS_DIR variable. Apart from a few unusual circumstances, handling of the sysroots-components directory should be automatic and the directory should not be referred to directly by recipes." * The SSTATE_SCAN_FILES entry is good, but it should mention that it is a space-separated list of filenames only, not paths, and standard wildcards (* and ?) are allowed. Typically recipes would add files they want to be scanned to the value of this variable rather than setting it outright, since there is a default specified in sstate.bbclass.
Paul, I've implemented all items listed in Comment 14. One follow up question - COMPONENTS_DIR is not currently defined in the glossary (referenced here: http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#structure-build-tmp-sysroots). Does it need to be? Thanks! Kristi
Yes we should - here's a definition: COMPONENTS_DIR: the directory where sysroot components are stored for each recipe. The OpenEmbedded build system uses these components when constructing recipe-specific sysroots for other recipes. The default is "${STAGING_DIR}-components" i.e. "${TMPDIR}/sysroots-components".
Thanks Paul, I've added the COMPONENTS_DIR variable to the glossary; http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#var-COMPONENTS_DIR Please review, and let me know if there are changes. -Kristi
Hi Kristi, I've reviewed the changes. * For the COMPONENTS_DIR entry, it seems like TMPDIR should be a link and there's a missing closing parenthesis but otherwise looks OK. * do_prepare_recipe_sysroot: you've written "i.e. WORKDIR" but it should be "i.e. recipe-sysroot and recipe-sysroot-native under ${WORKDIR}", and the closing parenthesis is missing. * SYSROOT_DESTDIR: typo "destidir" should be "destdir", and I missed out double quotes around the default value in my original text, could you also add those? There's also a copy-pasted default value from the above entry, I'd suggest just dropping that rather than correcting it as we've already mentioned the default. Thanks, Paul
Thank you for your detailed review Paul! I've implemented and corrected based on your notes in Comment 18. * COMPONENTS_DIR: http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#var-COMPONENTS_DIR * do_prepare_recipe_sysroot: http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#ref-tasks-prepare_recipe_sysroot * SYSROOT_DESTDIR http://www.yoctoproject.org/docs/2.4/ref-manual/ref-manual.html#var-SYSROOT_DESTDIR Let me know if any additional changes are needed. Thanks again! -Kristi
No feedback in a month, assuming this change works. Setting doc flag to "Done" and marking RESOLVED.