10248 2016-09-08 21:43:21 +0000 Suggested new section on how automatically added runtime dependencies work 2016-09-21 20:35:32 +0000 1 1 9 Documentation Reference handbook 2.2 x86 Multiple RESOLVED FIXED 21 Sept 2016: RESOLVED Medium normal 2.2 M4 1 ulfalizer srifenbark bluelightning poky.doc.watcher poky.watcher ross.burton srifenbark --- --- Done (doc changes complete) oldest_to_newest 65956 0 ulfalizer 2016-09-08 21:43:21 +0000 Hello Scott, Here's another largeish change. Can't think of any other ones I want to make for now. :) The description of the automatic runtime dependency mechanisms ('shlibdeps' and 'pcdeps') is very terse and leaves out important details (most importantly, where the information comes from). Instead of bloating the RDEPENDS glossary entry further, I think we should add a new section on automatically added runtime dependencies. I suggest the following changes: 1. Add a new section: 4.4. Automatically Added Runtime Dependencies The build system automatically adds common types of runtime dependencies between packages, meaning they do not need to be declared explicitly with [RDEPENDS]. There are two automatic mechanisms that deal, respectively, with shared libraries and pkg-config modules. Other types of runtime dependencies must still be declared manually. More information on the two automatic mechanisms is given below: * shlibdeps: During the [do_package] task of each recipe, all shared libraries installed by the recipe are located. For each shared library, the package that contains the shared library is registered as providing the shared library (more specifically, it is registered as providing the [soname](https://en.wikipedia.org/wiki/Soname) of the library). The resulting package-to-shared-library mapping is saved globally in [PKGDATA_DIR] by [do_packagedata]. Simultaneously, all executables and shared libraries installed by the recipe are inspected to see what shared libraries they link against. For each shared library dependency that is found, PKGDATA_DIR is queried to see if some package (likely from a different recipe) contains the shared library. If such a package is found, a runtime dependency is added from the package that depends on the shared library to the package that contains it. If the library being linked against is versioned (includes a version number), then the automatically added runtime dependency will specify at least that version, as if "<package> (>= <version>)" had been added to RDEPENDS. This forces an upgrade of the package that provides the shared library when installing the package that depends on it, if needed. If you want to avoid a package being registered as providing a particular shared library (e.g. because the library is for internal use only), then add the library to [PRIVATE_LIBS] inside the package's recipe. * pcdeps: During the [do_package] task of each recipe, all pkg-config modules (.pc files) installed by the recipe are located. For each module, the package that contains the module is registered as providing the module. The resulting package-to-module mapping is saved globally in [PKGDATA_DIR] by [do_packagedata]. Simultaneously, all pkg-config modules installed by the recipe are inspected to see what other modules they depend on. A module is seen as depending on another module if it contains a "Requires:" line that specifies the other module. For each module dependency, PKGDATA_DIR is queried to see if some package contains the module. If such a package is found, a runtime dependency is added from the package that depends on the module to the package that contains it. This mechanism most often infers dependencies between -dev packages. The do_package task depends on the [do_packagedata] task of each recipe in [DEPENDS] (via a [vardeps](https://www.yoctoproject.org/docs/2.2/bitbake-user-manual/bitbake-user-manual.html#variable-flags) declaration), which guarantees that the required package-to-shared-library/module mapping information will be available when needed (provided that DEPENDS has been set correctly). 2. Replace the beginning of the RDEPENDS glossary entry, before the paragraph that starts with "the names of the packages...", with the following. I rephrased it a bit to talk about the most important stuff first (the package runtime dependencies, rather than the do_build->do_package_write_* dependencies). RDEPENDS Lists runtime dependencies of a package. These are other packages that must be installed in order for the package to function correctly. As an example, the following assignment declares that the package foo needs the packages bar and baz to be installed: RDEPENDS_foo = "bar baz" The most common types of package runtime dependencies are detected and added automatically. Therefore, most recipes do not need to set RDEPENDS. See [4.3. Automatically Added Runtime Dependencies] for more information. The practical effect of the above RDEPENDS assignment is that bar and baz will be declared as dependencies inside the package foo when it is written out by one of the [do_package_write_*](http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-package_write_deb) tasks. Exactly how this is done depends on the package format used (see [PACKAGE_CLASSES]). When the corresponding package manager installs the package, it will know to also install the packages it depends on. To ensure that the packages bar and baz get built, the above RDEPENDS assignment also causes a task dependency to be added. This dependency is from the recipe's [do_build](not to be confused with [do_compile]) task to the do_package_write_* task of the recipes that build bar and baz. 3. Remove the following part of the RDEPENDS glossary entry, since it's covered by the new section, which is also linked earlier: In many cases you do not need to explicitly add runtime dependencies using RDEPENDS since some automatic handling occurs: shlibdeps: ... pcdeps: ... 4. (Unrelated clarification) Replace Because the RDEPENDS variable applies to packages being built, you should always use the variable in a form with an attached package name. with Because the RDEPENDS variable applies to packages being built, you should always use the variable in a form with an attached package name (remember that a single recipe can build multiple packages). 5. (Unrelated clarification) Replace The package name you attach to the RDEPENDS variable must appear as it would in the PACKAGES namespace before any renaming of the output package by classes like debian. with The package names you use with RDEPENDS must appear as they would in the PACKAGES variable. The [PKG] variable allows a different name to be used for the final package (for example, the [debian] class uses this to rename packages), but this final package name can't be used with RDEPENDS. This makes sense as RDEPENDS is meant to be independent of the package format used. 6. In the DEPENDS glossary entry, replace However, as explained in the RDEPENDS glossary entry, runtime dependencies will often be added automatically, meaning DEPENDS alone is sufficient for most recipes. with However, as explained in [4.3. Automatically Added Runtime Dependencies], runtime dependencies will often be added automatically, meaning DEPENDS alone is sufficient for most recipes. 7. Add the following to the end of the do_package glossary entry: The do_package task, in conjunction with [do_packagedata], also saves some important metadata about the package. See [PKGDESTWORK] and [4.3. Automatically Added Runtime Dependencies]. 8. Add the following to the end of the do_packagedata glossary entry: The data is saved in [PKGDATA_DIR]. 9. Add the following to the end of the PKGDATA_DIR: For examples of how this data is used, see [4.3. Automatically Added Runtime Dependencies] and [2.3.3. Viewing Package Information with oe-pkgdata-util]. Cheers, Ulf 65962 1 ulfalizer 2016-09-08 23:03:59 +0000 Self-nit: These three, The resulting package-to-shared-library mapping... The resulting package-to-module mapping... ...the required package-to-shared-library/module mapping... should be The resulting shared-library-to-package mapping... The resulting module-to-package mapping... ...the required shared-library/module-to-package mapping... 65963 2 ulfalizer 2016-09-08 23:46:20 +0000 One last nit: The following could be added to the end of the PRIVATE_LIBS entry: See [4.3. Automatically Added Runtime Dependencies] for more information. 66094 3 srifenbark 2016-09-13 18:25:53 +0000 Ulf, See the following: 1. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies - the new section. 2 - 5. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS - All the changes in the RDEPENDS glossary entry. 6. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-DEPENDS - see the middle bullet in the notes box for the change. 7. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-package - for the do_package task description. 8. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-packagedata - for the do_packagedata task description. 9. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PKGDATA_DIR - for the PKGDATA_DIR glossary change. See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PRIVATE_LIBS for the last nit in the comment list. 66099 4 ulfalizer 2016-09-13 19:10:51 +0000 Thanks! Some proofreading below: * This sounds a bit weird: Simultaneously, all pkg-config modules installed by the recipe are inspected to see what other modules on which the pkg-config modules depend. Maybe it should have been Simultaneously, all pkg-config modules installed by the recipe are inspected to see what other pkg-config modules they depend on. * PACKAGE_CLASSES just lists the package format that should be used, so maybe Exactly how this is done depends on the package format used used as described in PACKAGE_CLASSES. would be better as Exactly how this is done depends on which package format is used, which is determined by PACKAGE_CLASSES. * I had a braino in my original version. Instead of [vardeps], it should be [deptask]. The same link can be used. * There might be an extra 'the' in (not to be confused with the do_compile) * Typo: "to the to the" :) Cheers, Ulf 66118 5 ulfalizer 2016-09-14 00:51:40 +0000 One more change: I just realized that the paragraph that starts with "if the library being linked against is versioned..." might be a bit misleading. It's the version of the package that matters, not the version number in the soname. From looking at the code in package.bbclass, it seems you'd have to try really really hard not to get a version restriction added (maybe it's even dead code), so we could probably skip that part too. The entire paragraph could be replaced with the following: The automatically added runtime dependency also includes a version restriction. This version restriction specifies that at least the current version of the package that provides the shared library must be used, as if "<package> (>= <package version>)" had been added to [RDEPENDS]. This forces an upgrade of the package containing the shared library when installing the package that depends on the library, if needed. Cheers, Ulf 66135 6 srifenbark 2016-09-14 12:59:55 +0000 (In reply to comment #4) > Thanks! > > Some proofreading below: > > * This sounds a bit weird: > > Simultaneously, all pkg-config modules installed by the recipe are > inspected to see what other modules on which the pkg-config modules > depend. > > Maybe it should have been > > Simultaneously, all pkg-config modules installed by the recipe are > inspected to see what other pkg-config modules they depend on. I have a fixation with not ending a sentence with a preposition like "on". I have researched it and it is perfectly allowable in these modern times. I still try to write things such that they don't end this way. However, I am not going to be anal about it :) See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies > > * PACKAGE_CLASSES just lists the package format that should be used, so maybe > > Exactly how this is done depends on the package format used used as > described in PACKAGE_CLASSES. > > would be better as > > Exactly how this is done depends on which package format is used, which > is > determined by PACKAGE_CLASSES. See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS > > * I had a braino in my original version. Instead of [vardeps], it should > be [deptask]. The same link can be used. See last paragraph in http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies > > * There might be an extra 'the' in > > (not to be confused with the do_compile) > See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS > * Typo: "to the to the" :) Same... see http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS > > Cheers, > Ulf (In reply to comment #5) > One more change: > > I just realized that the paragraph that starts with "if the library being > linked against is versioned..." might be a bit misleading. It's the version > of the package that matters, not the version number in the soname. From > looking at the code in package.bbclass, it seems you'd have to try really > really hard not to get a version restriction added (maybe it's even dead > code), so we could probably skip that part too. > > The entire paragraph could be replaced with the following: > > The automatically added runtime dependency also includes a version > restriction. This version restriction specifies that at least the current > version of the package that provides the shared library must be used, as if > "<package> (>= <package version>)" had been added to [RDEPENDS]. This > forces > an upgrade of the package containing the shared library when installing the > package that depends on the library, if needed. Sounds good... See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies > > > Cheers, > Ulf 66152 7 ulfalizer 2016-09-14 18:52:34 +0000 (In reply to comment #6) > (In reply to comment #4) > > Thanks! > > > > Some proofreading below: > > > > * This sounds a bit weird: > > > > Simultaneously, all pkg-config modules installed by the recipe are > > inspected to see what other modules on which the pkg-config modules > > depend. > > > > Maybe it should have been > > > > Simultaneously, all pkg-config modules installed by the recipe are > > inspected to see what other pkg-config modules they depend on. > > I have a fixation with not ending a sentence with a preposition like "on". > I have researched it and it is perfectly allowable in these modern times. I > still try to write things such that they don't end this way. However, I am > not going to be anal about it :) > > See > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual. > html#automatically-added-runtime-dependencies Looks good to me. I only remembered that (fake?) Churchill quote (http://public.wsu.edu/~brians/errors/churchill.html), but I couldn't remember what it was about ("about what it was?" :P) Seems like a pretty arbitrary rule/trend, but I guess that goes for a lot of stuff in languages. > > > > * PACKAGE_CLASSES just lists the package format that should be used, so maybe > > > > Exactly how this is done depends on the package format used used as > > described in PACKAGE_CLASSES. > > > > would be better as > > > > Exactly how this is done depends on which package format is used, which > > is > > determined by PACKAGE_CLASSES. > > See > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS LGTM > > > > * I had a braino in my original version. Instead of [vardeps], it should > > be [deptask]. The same link can be used. > > See last paragraph in > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual. > html#automatically-added-runtime-dependencies LGTM > > > > * There might be an extra 'the' in > > > > (not to be confused with the do_compile) > > > > See > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS LGTM > > * Typo: "to the to the" :) > > Same... see > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS LGTM > > > > Cheers, > > Ulf > > (In reply to comment #5) > > One more change: > > > > I just realized that the paragraph that starts with "if the library being > > linked against is versioned..." might be a bit misleading. It's the version > > of the package that matters, not the version number in the soname. From > > looking at the code in package.bbclass, it seems you'd have to try really > > really hard not to get a version restriction added (maybe it's even dead > > code), so we could probably skip that part too. > > > > The entire paragraph could be replaced with the following: > > > > The automatically added runtime dependency also includes a version > > restriction. This version restriction specifies that at least the current > > version of the package that provides the shared library must be used, as if > > "<package> (>= <package version>)" had been added to [RDEPENDS]. This > > forces > > an upgrade of the package containing the shared library when installing the > > package that depends on the library, if needed. > > Sounds good... See > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual. > html#automatically-added-runtime-dependencies LGTM Thanks, Ulf 66153 8 ulfalizer 2016-09-14 20:38:08 +0000 Excuse the piggybacking, but I found two small semirelated documentation issues. I'll open a separate bug if I find more stuff so that we can close this one. :P 1. The description of PKGDESTWORK in the glossary looks wrong. I suspect do_package and do_packagedata got swapped here (or maybe "from" was supposed to be "for"): Points to a temporary work area used by the do_package task to write output from the do_packagedata task. Here's an improved version of the entire glossary entry: Points to a temporary work area where the [do_package] task saves package metadata. The PKGDESTWORK location defaults to the following: ${WORKDIR}/pkgdata Do not change this default. The [do_packagedata] task copies the package metadata from PKGDESTWORK to [PKGDATA_DIR] to make it available globally. 2. This description would be a more accurate replacement for the current version of '8.1.19. do_packagedata': Saves package metadata generated by the [do_package] task in [PKGDATA_DIR] to make it available globally. 66260 9 srifenbark 2016-09-16 17:42:53 +0000 (In reply to comment #8) > Excuse the piggybacking, but I found two small semirelated documentation > issues. I'll open a separate bug if I find more stuff so that we can close > this one. :P > > 1. The description of PKGDESTWORK in the glossary looks wrong. I suspect > do_package and do_packagedata got swapped here (or maybe "from" was > supposed > to be "for"): > > Points to a temporary work area used by the do_package task to write > output from the do_packagedata task. > > Here's an improved version of the entire glossary entry: > > Points to a temporary work area where the [do_package] task saves > package > metadata. The PKGDESTWORK location defaults to the following: > > ${WORKDIR}/pkgdata > > Do not change this default. > > The [do_packagedata] task copies the package metadata from PKGDESTWORK > to > [PKGDATA_DIR] to make it available globally. See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PKGDESTWORK for the change Ulf. > > > 2. This description would be a more accurate replacement for > the current version of '8.1.19. do_packagedata': > > Saves package metadata generated by the [do_package] task in > [PKGDATA_DIR] > to make it available globally. See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-packagedata 66268 10 ulfalizer 2016-09-16 20:09:42 +0000 (In reply to comment #9) > (In reply to comment #8) > > Excuse the piggybacking, but I found two small semirelated documentation > > issues. I'll open a separate bug if I find more stuff so that we can close > > this one. :P > > > > 1. The description of PKGDESTWORK in the glossary looks wrong. I suspect > > do_package and do_packagedata got swapped here (or maybe "from" was > > supposed > > to be "for"): > > > > Points to a temporary work area used by the do_package task to write > > output from the do_packagedata task. > > > > Here's an improved version of the entire glossary entry: > > > > Points to a temporary work area where the [do_package] task saves > > package > > metadata. The PKGDESTWORK location defaults to the following: > > > > ${WORKDIR}/pkgdata > > > > Do not change this default. > > > > The [do_packagedata] task copies the package metadata from PKGDESTWORK > > to > > [PKGDATA_DIR] to make it available globally. > > See > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var- > PKGDESTWORK for the change Ulf. > > > > > > > 2. This description would be a more accurate replacement for > > the current version of '8.1.19. do_packagedata': > > > > Saves package metadata generated by the [do_package] task in > > [PKGDATA_DIR] > > to make it available globally. > > See > http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks- > packagedata Looks good to me! 66435 11 srifenbark 2016-09-21 20:35:32 +0000 Thanks, Closing this out to RESOLVED and setting the doc flag to "done". Scott