Bug 5600

Enhancement - moving to 1.7.

Changing to Future.

I am adding some design notes here for when we get to this bug.  It is likely not to be addressed until post 1.7:

For the YP manual side:

 * I need to isolate a short summary for each variable.  The summary
   should be a single sentence and it will be appropriate for the 
   documentation.conf file.
 
 * The summary does not have to print in the manual.  In fact, I don't 
   want it to.  The variable descriptions are fine as they are right now.
   
 * The summary needs to be wrapped in comment or DocBook ID string such 
   that an outside script can easily find the string by processing the 
   ref-variables.xml file.

 * For each variable, I also need to somehow mark the longer description.
   I suspect by creating an invisible wrapper that an outside script can
   look for and then pull the text from within.  Might be able to just use
   some existing DocBook tags.  Need to figure out best method.

For the Script side (engineering)

 * Need to create a script that will process the ref-variables.xml file
   and extract the summary and a longer description for each variable.  The
   summary will be used in the documentation.conf file so that it does not
   have to be hand-edited.  

 * Need the script to also extract the longer description.  I am not exactly
   sure what needs to be done with the longer description but the script
   needs to be able to get to it.

> 
>  * Need the script to also extract the longer description.  I am not exactly
>    sure what needs to be done with the longer description but the script
>    needs to be able to get to it.

We are not using the long description in Toaster right now, but if we are going to structure the variable descriptions into a short description, a long description and context information we might as well do it in such a way that any of the three can be retrieved.

Hi,

The ref-variables.xml file now contains <info></info> tag pairs for each documented variable.  These tag pairs contain the short description of the variable that is used for the meta/conf/documentation.conf file variable descriptions.  The text in the tag pair is ignored by the manual's build process so it does not appear in the HTML or PDF version of the manual.

There are some gaps currently between the documentation.conf file and the ref-variables.xml file.  First gap is that the manual contains some variables that are not in the documentation.conf file.  Now that the <info></info> tags are in place, the scripting used by engineering should close this gap.  The second gap is there are several variables in the documentation.conf file that are not documented in the manual (list at the end of this note).  I have sent an email to Paul Eggleton that has the list and I can document these as needed.

One other small item is that one of the descriptions used in the documentation.conf file uses the literal & character, which breaks the manual make process.  I substituted the word "ampersand" for the character in the ref-variables.xml file.  You might consider some extra processing of the file to swap the character back in there for the documentation.conf file.

Here are the missing variables:

ASSUME_SHLIBS[doc] = "List of shlib:package[_version] mappings. Useful for lib packages in ASSUME_PROVIDED, for which automatic shlib dependency tracking does not work."

BBINCLUDELOGS_LINES[doc] = "Amount of log lines printed on failure."

BUILD_ARCH[doc] = "The name of the building architecture (e.g. i686)."

BUILD_OS[doc] = "The operating system (in lower case) of the building architecture (e.g. linux)."

CACHE[doc]  = "The directory holding the cache of the metadata."

CLEANBROKEN[doc] = "Specifies if 'make clean' does not work for a recipe (and therefore the build system should not try to use it during do_configure)"

CONF_VERSION[doc] = "Tracks the version of local.conf.  Increased each time build/conf/ changes incompatibly."

CVSDIR[doc] = "The directory where cvs checkouts will be stored in."

DATE[doc] = "The date the build was started using YMD format."

FONT_EXTRA_RDEPENDS[doc] = "When a recipe inherits the fontcache class, this variable specifies runtime dependencies for font packages. This variable defaults to 'fontconfig-utils'."

GITDIR[doc] = "The directory where Git clones will be stored."

HOST_ARCH[doc] = "The name of the target architecture. Normally same as the 

TARGET_ARCH."

HOST_OS[doc] = "The name of the target operating system. Normally the same as the TARGET_OS."

HOST_PREFIX[doc] = "The prefix for the cross compile toolchain. Normally same as the TARGET_PREFIX."

HOST_VENDOR[doc] = "The name of the vendor. Normally same as the TARGET_VENDOR."

PACKAGE_INSTALL_ATTEMPTONLY[doc] = "List of packages attempted to be installed. If a listed package fails to install, the build system does not generate an error. This variable is generally not user-defined."

PRIORITY[doc] = "Indicates the importance of a package.  The default value is 'optional'.  Other standard values are 'required', 'standard' and 'extra'."

SOURCE_MIRROR_FETCH[doc] = "Switch marking build as source fetcher. Used to skip COMPATIBLE_* checking."

SVNDIR[doc] = "The directory where Subversion checkouts will be stored."

TARGET_PREFIX[doc] = "The prefix for the cross-compile toolchain (e.g. arm-linux-)."

TARGET_SYS[doc] = "The target system is comprised of TARGET_ARCH,TARGET_VENDOR and TARGET_OS."

TIME[doc] = "The time the build was started using HMS format."

USE_DEVFS[doc] = "When building images, specifies to populate or not /dev. This variable defaults to '1' (leave directory empty, surely because devtmpfs do the job). Set it to '0' to use makedevs (or consider using a custom file with 

IMAGE_DEVICE_TABLES)."

Paul Eggleton has created a patch that I applied to the documentation/tools folder.  The patch introduced a new Python script (update-documentation-conf) that when run creates a patch for the documentation.conf file.  Once the patch is created, the developer would have to submit the patch to oe-core to have it applied.  This effectively syncs the hidden, brief descriptions in the ref-manual variables glossary file with the same descriptions used in the documentation.conf file.  For now, it is a hand process of sorts.  Future plans could be to automate the process such that any change to strings in the variables chapter would automatically update the same strings in the documentation.conf file.

Scott

CLEANBROKEN has just recently been added to the ref manual, but for the rest yes we should add them. For HOST_*, BUILD_* and TARGET_* we would want to use similar wording to the existing variables we do have documented (e.g. HOST_SYS).

Hi, 

I am setting this bug to review.  I have all the variables in the ref-manual glossary and with the <note></note> brief descriptions in.  I believe Paul has added scripting to allow a user of Yocto to automatically update the documentation.conf file with descriptions that will match those in the manual.

So unless there are more variables we are missing, this issue should be pretty much taken care of.  I will leave it in REVIEW for a while.  For any additional variables needed, we can get those in outside of the context of this particular bug. 

Scott

How do we verify that this works? Do we run the script, apply the patch to documentation.conf and check what happens in Toaster?

In terms of the actual content we're currently waiting for me to run the script and send a patch. I haven't done that yet because I still need to run through some of the descriptions with Scott.

The variables have passed through a review and can be viewed again if you want to.  I will leave this up a while as IN PROGRESS REVIEW.  Once some time passes with no comments, I will close.

To view the latest description of any of these variables in the comments, use the http://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#var-<variable_name> (e.g. http://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#var-ASSUME_SHLIB.

Thanks,
Scott

I think we should test this by running the script then review the variable descriptions in Toaster. This is mainly because inserting text automatically into an interface can cause issues related to the content length. 

I'll be happy to run the test myself if Paul can give me some instructions on how to run the script.

I don't know where we stand on this.  

Scott

When I did the sync I found there were some issues with some of the short descriptions coming from the manual. I need to review these with you and I suspect that will have to wait until 1.9 as well.

I haven't got any updates on the changes here yet.  I am setting the days to 1.0 to get it off the list.

So no activity on this for over two years... Are we good to close this?

Scott

Have the inconsistencies between the short descriptions in the doc source and documentation.conf been addressed? It doesn't seem so at least based on the comments here.

Setting to IN PROGRESS DESIGN.  

After the 2.4 release, I will run the yocto-docs/documentation/tools/update-documentation-conf script:

   $ cd yocto-docs/documentation
   $ chmod 755 tools/update-documentation-conf
   $ ./update-documentation-conf ~/poky

This will update the ~/poky/meta/conf/documentation.conf file so that all the strings match the [doc] strings used in the ref-manual/ref-variables.xml file.  With this information, I will scrub the messages to make sure they are reasonable.  Then, after approval, I can run the script again to sync the poky/meta/conf/documentation.conf file with the updated [doc] strings from the manual.  After that, we can submit the diff file as a patch for poky/meta/conf/documentation.conf.

Scott

Paul, I tried running this and it didn't work for me. I received the following output: 
klynn@samsung:~/yocto-docs/documentation/tools$ ./update-documentation-conf ~/poky
Traceback (most recent call last):
  File "./update-documentation-conf", line 27, in <module>
    from lxml import etree
ImportError: No module named lxml

What do I need to do to fix this? Thanks! 
Kristi

Kristi - it appears you need to install python-lxml on the machine you're running the script on, that should fix the error.

Thanks Paul! I was able to run the script with your fix. A heads up - I am going to do a scrub on the variable descriptions for grammar and that they read well. 

After I make changes I'll let you know so you can review before I re-run the script.  

Kristi

Hi Paul - While going through the variable descriptions in the documentation.conf file, I've run into a situation where there are entries for variables in that file that are not documented in the ref-manual variable section. I thought all these entries were auto-generated out of that ref-variables.xml file, which is part of the ref-manual. Apparently not, where are these others coming from? 

Thanks! 
Kristi

Kristi - that is where we would like to get to, but where we started was that documentation.conf existed first, so there are some descriptions in there that we don't have in the manual - we need to add those to the manual.

Additionally we have (or used to have, I'm not sure if they have been corrected) some short descriptions in the manual that are too verbose for documentation.conf. I can help work through those but we do need to fix them or we can't start using the sync script.

I suggest to close this bug as the documentation is not designed to have short descriptions of each variable.

Before doing this, we could make sure that no variable is missing from documentation.conf, provided such information is still in use by Toaster or is used elsewhere in Bitbake / OE. See my question on https://lore.kernel.org/openembedded-core/9e413386-3fb8-22ca-3a4f-7630b382e9d8@bootlin.com/T/#u

Bulk move of 5.1 M4 to 5.2 M2 as approved by AlexB.

Bulk move of bugs I am assigned to from Milestone 5.2M4 to 5.3M1.