Bug 7134

Summary: ENTITY variables in poky.ent not expanding consistently in HTML and PDF versions
Product: [Documentation] General Docs Reporter: Scott Rifenbark <srifenbark>
Component: docs-generalAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact: Libertad <libertad.gonzalez.de.la.cruz>
Severity: critical    
Priority: High CC: alexandru.c.georgescu, bluelightning, jessica.zhang, jose.perez.carranza, richard.purdie, srifenbark
Version: unspecified   
Target Milestone: 1.8 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Done (doc changes complete)
Attachments:
Description Flags
grep results for &DISTRO; throughout the manual set XML files
none
Explanation for &DISTRO; expansions (WORKED and FAILED)
none
poky.ent file with all the ENTITY declarations. none

Description Scott Rifenbark 2015-01-13 17:30:04 UTC 
Created attachment 2318 [details]
grep results for &DISTRO; throughout the manual set XML files

Currently, the YP manuals employ a documentation/poky.ent file that defines many ENTITY variables.  During the transformation from XML files to both HTML and PDF output, these variables should consistently expand to their literal values for display in the HTML and PDF versions of the manuals.  This scheme worked in the past but is now broken.

Prior to working on the 1.8 release, my development system was a Ubuntu 12.04 distribution running on a Lenovo X230 Thinkpad.  The documentation is built using DocBook 4.2 with the "current" XSL stylesheets (e.g. http://docbook.sourceforge.net/release/xsl/current/xhtml/docbook.xsl), which are imported into each manual's customization layer file (e.g. adt-customization.xsl).

After upgrading my distribution to Ubuntu 14.10 I began experiencing erratic variable expansions.  Some variables would correctly expand and appear as strings in the HTML and PDF output, while occurrences of the same variable would not expand. The result causes dropped critical information in the manual - for example missing &DISTRO; (e.g. 1.8) strings in pathnames used in the documentation.

The issue is not limited to the 1.8 versions of the manuals but surfaces for all releases now.  So, cloning a "dizzy" branch and building the manuals results in the same issues.  I have made sure my system is up-to-date with all packages and have the packages installed as described in the YP reference manual at http://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#ubuntu-packages.  

To reproduce the behavior do the following:

1. git clone http://git.yoctoproject.org/git/yocto-docs
2. cd ~/yocto-docs/documentation
3. make html DOC=adt-manual

This will create a ~/yocto-docs/documentation/adt-manual/adt-manual.html file.  Opening this file in your browser displays the adt-manual.

To show the weirdness here, I searched for the '&DISTRO;' string, which is the first ENTITY variable in the ~/yocto-docs/documentation/poky.ent file while in the ~/yocto-docs/documentation directory:

1. cd ~/yocto-docs/documentation
2. git grep "&DISTRO;" *

This search reveals all the occurrences of that variable in the entire documentation set.  I then went through the affected manuals for that particular ENTITY variable and noted whether they expanded or not.  I am including two files to show the results: grep-results and entity-errors.  The grep-results shows each occurrence of the &DISTRO; variable in all the XML source files.  The entity-errors file is my listing of each found result followed by what actually appears in the HTML version of the manual.  Some work and some don't.  I am also including the ~/yocto-docs/documentation/poky.ent file as well.

It is possible to work-around this by not using the poky.ent file and simply hand-supplying the proper strings.  This greatly increases the steps to produce each release of the documentation, however, and introduces the possibility of errors.

This is likely something simple but I have spent too much time on it already so I am hoping someone can help.

Scott
Comment 1 Scott Rifenbark 2015-01-13 17:31:24 UTC 
Created attachment 2319 [details]
Explanation for &DISTRO; expansions (WORKED and FAILED)
Comment 2 Scott Rifenbark 2015-01-13 17:32:00 UTC 
Created attachment 2320 [details]
poky.ent file with all the ENTITY declarations.
Comment 3 Scott Rifenbark 2015-01-13 17:33:39 UTC 
adding self to CC list and others
Comment 4 Richard Purdie 2015-01-14 17:46:35 UTC 
I can confirm I can reproduce this on Ubuntu 14.10 myself. I tried building libxslt 1.1.26, the version in Ubuntu 12.04 and using that, the same problem exists there. its therefore likely something in one of the dependencies of libxslt such as libxml2 which is causing the issue.
Comment 5 Richard Purdie 2015-01-14 18:07:32 UTC 
Additionally, reverting to libxml 2.7.8 fixes the generated document so it looks like a libxml2 issue.
Comment 6 Richard Purdie 2015-01-14 19:01:34 UTC 
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=93739f10bb026a7ec8f1a3d21c309980471c36bd and the preceeding commit have my debug info so far
Comment 7 Richard Purdie 2015-01-15 09:31:00 UTC 
The issue is caused by the CVE-2014-3660 fix applied to libxml2. Its fixed in master of libxml2 but not in any release. The fix is:

https://git.gnome.org/browse/libxml2/commit/?id=72a46a519ce7326d9a00f0b6a7f2a8e958cd1675

We've seen such differences in testing as some people have fixed CVE, some have not. We should integrate the CVE into Yocto, then give Scott instructions on how to use the Yocto built xsltproc command until the fix makes it into the various distros.
Comment 8 Richard Purdie 2015-01-15 15:19:55 UTC 
Just to capture the commit log for history:
Add to local.conf: 
PREFERRED_VERSION_libxslt-native = "1.1.26" 
PREFERRED_VERSION_libxml2-native = "2.7.8" 

Reproducer for doc bug: 
cd documentation/adt-manual 
/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml

With bug: grep /opt/ * -r | grep env 
adt-command.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux 
adt-manual.html: /opt/poky//environment-setup-x86_64-poky-linux 
adt-manual.html: $ source /opt/poky//environment-setup-i586-poky-linux 
adt-manual.html: $ source /opt/poky/1.8/environment-setup-i586-poky-linux 
adt-prepare.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux 

(note missing 1.8) 

without bug: $ grep /opt/ * -r | grep env 
adt-command.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux 
adt-manual.html: /opt/poky/1.8/environment-setup-x86_64-poky-linux 
adt-manual.html: $ source /opt/poky/1.8/environment-setup-i586-poky-linux 
adt-manual.html: $ source /opt/poky/1.8/environment-setup-i586-poky-linux 
adt-prepare.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux 

(note 1.8 present)
Comment 9 Richard Purdie 2015-01-15 15:22:45 UTC 
So to work around this issue, we can use the build system to help us. Setup a checkout of poky somewhere and:

bitbake libxslt-native

then

export PATH=<path to buliddir>/tmp/sysroots/x86_64-linux/usr/bin:$PATH

then run doc make commands as usual. If you change to a new shell, you need to run the export PATH command again.

You need a branch of poky with the fix in it, currently that is master-next but it will merge to master soon.

Once the upstream distros update their packages to include the fix to libxml2, this workaround will no longer be needed.
Comment 11 Paul Eggleton 2015-01-16 11:22:51 UTC 
Finding information about resolution of this issue in the various distributions is distressingly difficult; despite being fixed upstream in October, the fix is only now trickling into some distributions, for others it's just not clear what the plans are for fixing it.

FWIW, it looks like OpenSuSE has a fix for this in their package source although I can't tell if it has actually been pushed out to package updates. Debian has a bug open but there aren't any comments from anyone other than the submitter at the moment:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768089

I couldn't find anything related for Fedora or Ubuntu.
Comment 12 Scott Rifenbark 2017-06-08 13:41:08 UTC 
Setting the documentation flag to "done" for this bug.

Scott