Bug 10248

Summary:	Suggested new section on how automatically added runtime dependencies work
Product:	[Documentation] Reference	Reporter:	ulfalizer
Component:	handbook	Assignee:	Scott Rifenbark <srifenbark>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium	CC:	bluelightning, poky.doc.watcher, poky.watcher, ross.burton, srifenbark
Version:	2.2
Target Milestone:	2.2 M4
Hardware:	x86
OS:	Multiple
Whiteboard:	21 Sept 2016: RESOLVED
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Done (doc changes complete)

Description ulfalizer 2016-09-08 21:43:21 UTC

Hello Scott,

Here's another largeish change. Can't think of any other ones I want to make
for now. :)

The description of the automatic runtime dependency mechanisms ('shlibdeps' and
'pcdeps') is very terse and leaves out important details (most importantly,
where the information comes from). Instead of bloating the RDEPENDS glossary
entry further, I think we should add a new section on automatically added
runtime dependencies. I suggest the following changes:

1. Add a new section:

  4.4. Automatically Added Runtime Dependencies
  
  The build system automatically adds common types of runtime dependencies
  between packages, meaning they do not need to be declared explicitly with
  [RDEPENDS]. There are two automatic mechanisms that deal, respectively, with
  shared libraries and pkg-config modules. Other types of runtime dependencies
  must still be declared manually.

  More information on the two automatic mechanisms is given below:
  
   * shlibdeps:
     During the [do_package] task of each recipe, all shared libraries
     installed by the recipe are located. For each shared library, the package
     that contains the shared library is registered as providing the shared
     library (more specifically, it is registered as providing the
     [soname](https://en.wikipedia.org/wiki/Soname) of the library). The
     resulting package-to-shared-library mapping is saved globally in
     [PKGDATA_DIR] by [do_packagedata].
  
     Simultaneously, all executables and shared libraries installed by the
     recipe are inspected to see what shared libraries they link against. For
     each shared library dependency that is found, PKGDATA_DIR is queried to
     see if some package (likely from a different recipe) contains the shared
     library. If such a package is found, a runtime dependency is added from
     the package that depends on the shared library to the package that
     contains it.
  
     If the library being linked against is versioned (includes a version
     number), then the automatically added runtime dependency will specify at
     least that version, as if "<package> (>= <version>)" had been added to
     RDEPENDS. This forces an upgrade of the package that provides the shared
     library when installing the package that depends on it, if needed.
  
     If you want to avoid a package being registered as providing a particular
     shared library (e.g. because the library is for internal use only), then
     add the library to [PRIVATE_LIBS] inside the package's recipe.
  
   * pcdeps:
     During the [do_package] task of each recipe, all pkg-config modules (.pc
     files) installed by the recipe are located. For each module, the package
     that contains the module is registered as providing the module. The
     resulting package-to-module mapping is saved globally in [PKGDATA_DIR] by
     [do_packagedata].
  
     Simultaneously, all pkg-config modules installed by the recipe are
     inspected to see what other modules they depend on. A module is seen as
     depending on another module if it contains a "Requires:" line that
     specifies the other module. For each module dependency, PKGDATA_DIR is
     queried to see if some package contains the module. If such a package is
     found, a runtime dependency is added from the package that depends on the
     module to the package that contains it.
  
     This mechanism most often infers dependencies between -dev packages.

  The do_package task depends on the [do_packagedata] task of each recipe in
  [DEPENDS] (via a
  [vardeps](https://www.yoctoproject.org/docs/2.2/bitbake-user-manual/bitbake-user-manual.html#variable-flags)
  declaration), which guarantees that the required
  package-to-shared-library/module mapping information will be available when
  needed (provided that DEPENDS has been set correctly).


2. Replace the beginning of the RDEPENDS glossary entry, before the paragraph
that starts with "the names of the packages...", with the following. I
rephrased it a bit to talk about the most important stuff first (the package
runtime dependencies, rather than the do_build->do_package_write_*
dependencies).

  RDEPENDS
  
  Lists runtime dependencies of a package. These are other packages that must
  be installed in order for the package to function correctly. As an example,
  the following assignment declares that the package foo needs the packages bar
  and baz to be installed:
  
    RDEPENDS_foo = "bar baz"
  
  The most common types of package runtime dependencies are detected and added
  automatically. Therefore, most recipes do not need to set RDEPENDS. See
  [4.3. Automatically Added Runtime Dependencies] for more information.
  
  The practical effect of the above RDEPENDS assignment is that bar and baz
  will be declared as dependencies inside the package foo when it is written
  out by one of the
  [do_package_write_*](http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-package_write_deb)
  tasks. Exactly how this is done depends on the package format used (see
  [PACKAGE_CLASSES]). When the corresponding package manager installs the
  package, it will know to also install the packages it depends on.
  
  To ensure that the packages bar and baz get built, the above RDEPENDS
  assignment also causes a task dependency to be added. This dependency is from
  the recipe's [do_build](not to be confused with [do_compile]) task to the
  do_package_write_* task of the recipes that build bar and baz.


3. Remove the following part of the RDEPENDS glossary entry, since it's covered
by the new section, which is also linked earlier:

  In many cases you do not need to explicitly add runtime dependencies using
  RDEPENDS since some automatic handling occurs:
  
    shlibdeps: ...
  
    pcdeps: ...


4. (Unrelated clarification)

Replace

  Because the RDEPENDS variable applies to packages being built, you should
  always use the variable in a form with an attached package name.

with

  Because the RDEPENDS variable applies to packages being built, you should
  always use the variable in a form with an attached package name (remember
  that a single recipe can build multiple packages).


5. (Unrelated clarification)

Replace

  The package name you attach to the RDEPENDS variable must appear as it would
  in the PACKAGES namespace before any renaming of the output package by
  classes like debian. 

with

  The package names you use with RDEPENDS must appear as they would in the
  PACKAGES variable. The [PKG] variable allows a different name to be used for
  the final package (for example, the [debian] class uses this to rename
  packages), but this final package name can't be used with RDEPENDS. This
  makes sense as RDEPENDS is meant to be independent of the package format
  used.


6. In the DEPENDS glossary entry, replace

  However, as explained in the RDEPENDS glossary entry, runtime dependencies
  will often be added automatically, meaning DEPENDS alone is sufficient for
  most recipes.

with

  However, as explained in [4.3. Automatically Added Runtime Dependencies],
  runtime dependencies will often be added automatically, meaning DEPENDS alone
  is sufficient for most recipes.


7. Add the following to the end of the do_package glossary entry:

  The do_package task, in conjunction with [do_packagedata], also saves some
  important metadata about the package. See [PKGDESTWORK] and
  [4.3. Automatically Added Runtime Dependencies].


8. Add the following to the end of the do_packagedata glossary entry:

  The data is saved in [PKGDATA_DIR].


9. Add the following to the end of the PKGDATA_DIR:

  For examples of how this data is used, see
  [4.3. Automatically Added Runtime Dependencies]
  and [2.3.3. Viewing Package Information with oe-pkgdata-util].


Cheers,
Ulf

Comment 1 ulfalizer 2016-09-08 23:03:59 UTC

Self-nit:

These three,

  The resulting package-to-shared-library mapping...
  The resulting package-to-module mapping...
  ...the required package-to-shared-library/module mapping...

should be

  The resulting shared-library-to-package mapping...
  The resulting module-to-package mapping...
  ...the required shared-library/module-to-package mapping...

Comment 2 ulfalizer 2016-09-08 23:46:20 UTC

One last nit:

The following could be added to the end of the PRIVATE_LIBS entry:

  See [4.3. Automatically Added Runtime Dependencies] for more
  information.

Comment 3 Scott Rifenbark 2016-09-13 18:25:53 UTC

Ulf, 

See the following:

1. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies - the new section.

2 - 5.  http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS  - All the changes in the RDEPENDS glossary entry.

6. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-DEPENDS - see the middle bullet in the notes box for the change.

7. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-package - for the do_package task description.

8. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-packagedata - for the do_packagedata task description.

9. http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PKGDATA_DIR - for the PKGDATA_DIR glossary change.

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PRIVATE_LIBS for the last nit in the comment list.

Comment 4 ulfalizer 2016-09-13 19:10:51 UTC

Thanks!

Some proofreading below:

* This sounds a bit weird:

    Simultaneously, all pkg-config modules installed by the recipe are
    inspected to see what other modules on which the pkg-config modules
    depend.

  Maybe it should have been

    Simultaneously, all pkg-config modules installed by the recipe are
    inspected to see what other pkg-config modules they depend on.

* PACKAGE_CLASSES just lists the package format that should be used, so maybe

    Exactly how this is done depends on the package format used used as
    described in PACKAGE_CLASSES.

  would be better as

    Exactly how this is done depends on which package format is used, which is
    determined by PACKAGE_CLASSES.

* I had a braino in my original version. Instead of [vardeps], it should
  be [deptask]. The same link can be used.

* There might be an extra 'the' in

    (not to be confused with the do_compile)

* Typo: "to the to the" :)

Cheers,
Ulf

Comment 5 ulfalizer 2016-09-14 00:51:40 UTC

One more change:

I just realized that the paragraph that starts with "if the library being linked against is versioned..." might be a bit misleading. It's the version of the package that matters, not the version number in the soname. From looking at the code in package.bbclass, it seems you'd have to try really really hard not to get a version restriction added (maybe it's even dead code), so we could probably skip that part too.

The entire paragraph could be replaced with the following:

  The automatically added runtime dependency also includes a version
  restriction. This version restriction specifies that at least the current
  version of the package that provides the shared library must be used, as if
  "<package> (>= <package version>)" had been added to [RDEPENDS]. This forces
  an upgrade of the package containing the shared library when installing the
  package that depends on the library, if needed.


Cheers,
Ulf

Comment 6 Scott Rifenbark 2016-09-14 12:59:55 UTC

(In reply to comment #4)
> Thanks!
> 
> Some proofreading below:
> 
> * This sounds a bit weird:
> 
>     Simultaneously, all pkg-config modules installed by the recipe are
>     inspected to see what other modules on which the pkg-config modules
>     depend.
> 
>   Maybe it should have been
> 
>     Simultaneously, all pkg-config modules installed by the recipe are
>     inspected to see what other pkg-config modules they depend on.

I have a fixation with not ending a sentence with a preposition like "on".  I have researched it and it is perfectly allowable in these modern times.  I still try to write things such that they don't end this way.  However, I am not going to be anal about it :)

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies

> 
> * PACKAGE_CLASSES just lists the package format that should be used, so maybe
> 
>     Exactly how this is done depends on the package format used used as
>     described in PACKAGE_CLASSES.
> 
>   would be better as
> 
>     Exactly how this is done depends on which package format is used, which
> is
>     determined by PACKAGE_CLASSES.

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS

> 
> * I had a braino in my original version. Instead of [vardeps], it should
>   be [deptask]. The same link can be used.

See last paragraph in http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies

> 
> * There might be an extra 'the' in
> 
>     (not to be confused with the do_compile)
> 

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS

> * Typo: "to the to the" :)

Same... see http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS

> 
> Cheers,
> Ulf

(In reply to comment #5)
> One more change:
> 
> I just realized that the paragraph that starts with "if the library being
> linked against is versioned..." might be a bit misleading. It's the version
> of the package that matters, not the version number in the soname. From
> looking at the code in package.bbclass, it seems you'd have to try really
> really hard not to get a version restriction added (maybe it's even dead
> code), so we could probably skip that part too.
> 
> The entire paragraph could be replaced with the following:
> 
>   The automatically added runtime dependency also includes a version
>   restriction. This version restriction specifies that at least the current
>   version of the package that provides the shared library must be used, as if
>   "<package> (>= <package version>)" had been added to [RDEPENDS]. This
> forces
>   an upgrade of the package containing the shared library when installing the
>   package that depends on the library, if needed.

Sounds good... See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#automatically-added-runtime-dependencies

> 
> 
> Cheers,
> Ulf

Comment 7 ulfalizer 2016-09-14 18:52:34 UTC

(In reply to comment #6)
> (In reply to comment #4)
> > Thanks!
> > 
> > Some proofreading below:
> > 
> > * This sounds a bit weird:
> > 
> >     Simultaneously, all pkg-config modules installed by the recipe are
> >     inspected to see what other modules on which the pkg-config modules
> >     depend.
> > 
> >   Maybe it should have been
> > 
> >     Simultaneously, all pkg-config modules installed by the recipe are
> >     inspected to see what other pkg-config modules they depend on.
> 
> I have a fixation with not ending a sentence with a preposition like "on". 
> I have researched it and it is perfectly allowable in these modern times.  I
> still try to write things such that they don't end this way.  However, I am
> not going to be anal about it :)
> 
> See
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.
> html#automatically-added-runtime-dependencies

Looks good to me.

I only remembered that (fake?) Churchill quote (http://public.wsu.edu/~brians/errors/churchill.html), but I couldn't remember what it was about ("about what it was?" :P)

Seems like a pretty arbitrary rule/trend, but I guess that goes for a lot of stuff in languages.
 
> > 
> > * PACKAGE_CLASSES just lists the package format that should be used, so maybe
> > 
> >     Exactly how this is done depends on the package format used used as
> >     described in PACKAGE_CLASSES.
> > 
> >   would be better as
> > 
> >     Exactly how this is done depends on which package format is used, which
> > is
> >     determined by PACKAGE_CLASSES.
> 
> See
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS

LGTM

> > 
> > * I had a braino in my original version. Instead of [vardeps], it should
> >   be [deptask]. The same link can be used.
> 
> See last paragraph in
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.
> html#automatically-added-runtime-dependencies

LGTM

> > 
> > * There might be an extra 'the' in
> > 
> >     (not to be confused with the do_compile)
> > 
> 
> See
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS

LGTM

> > * Typo: "to the to the" :)
> 
> Same... see
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-RDEPENDS

LGTM

> > 
> > Cheers,
> > Ulf
> 
> (In reply to comment #5)
> > One more change:
> > 
> > I just realized that the paragraph that starts with "if the library being
> > linked against is versioned..." might be a bit misleading. It's the version
> > of the package that matters, not the version number in the soname. From
> > looking at the code in package.bbclass, it seems you'd have to try really
> > really hard not to get a version restriction added (maybe it's even dead
> > code), so we could probably skip that part too.
> > 
> > The entire paragraph could be replaced with the following:
> > 
> >   The automatically added runtime dependency also includes a version
> >   restriction. This version restriction specifies that at least the current
> >   version of the package that provides the shared library must be used, as if
> >   "<package> (>= <package version>)" had been added to [RDEPENDS]. This
> > forces
> >   an upgrade of the package containing the shared library when installing the
> >   package that depends on the library, if needed.
> 
> Sounds good... See
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.
> html#automatically-added-runtime-dependencies

LGTM

Thanks,
Ulf

Comment 8 ulfalizer 2016-09-14 20:38:08 UTC

Excuse the piggybacking, but I found two small semirelated documentation
issues. I'll open a separate bug if I find more stuff so that we can close
this one. :P

1. The description of PKGDESTWORK in the glossary looks wrong. I suspect
   do_package and do_packagedata got swapped here (or maybe "from" was supposed
   to be "for"):

     Points to a temporary work area used by the do_package task to write
     output from the do_packagedata task.

   Here's an improved version of the entire glossary entry:

     Points to a temporary work area where the [do_package] task saves package
     metadata. The PKGDESTWORK location defaults to the following:

       ${WORKDIR}/pkgdata

     Do not change this default.

     The [do_packagedata] task copies the package metadata from PKGDESTWORK to
     [PKGDATA_DIR] to make it available globally.


2. This description would be a more accurate replacement for
   the current version of '8.1.19. do_packagedata':

     Saves package metadata generated by the [do_package] task in [PKGDATA_DIR]
     to make it available globally.

Comment 9 Scott Rifenbark 2016-09-16 17:42:53 UTC

(In reply to comment #8)
> Excuse the piggybacking, but I found two small semirelated documentation
> issues. I'll open a separate bug if I find more stuff so that we can close
> this one. :P
> 
> 1. The description of PKGDESTWORK in the glossary looks wrong. I suspect
>    do_package and do_packagedata got swapped here (or maybe "from" was
> supposed
>    to be "for"):
> 
>      Points to a temporary work area used by the do_package task to write
>      output from the do_packagedata task.
> 
>    Here's an improved version of the entire glossary entry:
> 
>      Points to a temporary work area where the [do_package] task saves
> package
>      metadata. The PKGDESTWORK location defaults to the following:
> 
>        ${WORKDIR}/pkgdata
> 
>      Do not change this default.
> 
>      The [do_packagedata] task copies the package metadata from PKGDESTWORK
> to
>      [PKGDATA_DIR] to make it available globally.

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PKGDESTWORK for the change Ulf.

> 
> 
> 2. This description would be a more accurate replacement for
>    the current version of '8.1.19. do_packagedata':
> 
>      Saves package metadata generated by the [do_package] task in
> [PKGDATA_DIR]
>      to make it available globally.

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-packagedata

Comment 10 ulfalizer 2016-09-16 20:09:42 UTC

(In reply to comment #9)
> (In reply to comment #8)
> > Excuse the piggybacking, but I found two small semirelated documentation
> > issues. I'll open a separate bug if I find more stuff so that we can close
> > this one. :P
> > 
> > 1. The description of PKGDESTWORK in the glossary looks wrong. I suspect
> >    do_package and do_packagedata got swapped here (or maybe "from" was
> > supposed
> >    to be "for"):
> > 
> >      Points to a temporary work area used by the do_package task to write
> >      output from the do_packagedata task.
> > 
> >    Here's an improved version of the entire glossary entry:
> > 
> >      Points to a temporary work area where the [do_package] task saves
> > package
> >      metadata. The PKGDESTWORK location defaults to the following:
> > 
> >        ${WORKDIR}/pkgdata
> > 
> >      Do not change this default.
> > 
> >      The [do_packagedata] task copies the package metadata from PKGDESTWORK
> > to
> >      [PKGDATA_DIR] to make it available globally.
> 
> See
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-
> PKGDESTWORK for the change Ulf.
> 
> > 
> > 
> > 2. This description would be a more accurate replacement for
> >    the current version of '8.1.19. do_packagedata':
> > 
> >      Saves package metadata generated by the [do_package] task in
> > [PKGDATA_DIR]
> >      to make it available globally.
> 
> See
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-tasks-
> packagedata

Looks good to me!

Comment 11 Scott Rifenbark 2016-09-21 20:35:32 UTC

Thanks, 

Closing this out to RESOLVED and setting the doc flag to "done".

Scott