Created attachment 2318 [details] grep results for &DISTRO; throughout the manual set XML files Currently, the YP manuals employ a documentation/poky.ent file that defines many ENTITY variables. During the transformation from XML files to both HTML and PDF output, these variables should consistently expand to their literal values for display in the HTML and PDF versions of the manuals. This scheme worked in the past but is now broken. Prior to working on the 1.8 release, my development system was a Ubuntu 12.04 distribution running on a Lenovo X230 Thinkpad. The documentation is built using DocBook 4.2 with the "current" XSL stylesheets (e.g. http://docbook.sourceforge.net/release/xsl/current/xhtml/docbook.xsl), which are imported into each manual's customization layer file (e.g. adt-customization.xsl). After upgrading my distribution to Ubuntu 14.10 I began experiencing erratic variable expansions. Some variables would correctly expand and appear as strings in the HTML and PDF output, while occurrences of the same variable would not expand. The result causes dropped critical information in the manual - for example missing &DISTRO; (e.g. 1.8) strings in pathnames used in the documentation. The issue is not limited to the 1.8 versions of the manuals but surfaces for all releases now. So, cloning a "dizzy" branch and building the manuals results in the same issues. I have made sure my system is up-to-date with all packages and have the packages installed as described in the YP reference manual at http://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#ubuntu-packages. To reproduce the behavior do the following: 1. git clone http://git.yoctoproject.org/git/yocto-docs 2. cd ~/yocto-docs/documentation 3. make html DOC=adt-manual This will create a ~/yocto-docs/documentation/adt-manual/adt-manual.html file. Opening this file in your browser displays the adt-manual. To show the weirdness here, I searched for the '&DISTRO;' string, which is the first ENTITY variable in the ~/yocto-docs/documentation/poky.ent file while in the ~/yocto-docs/documentation directory: 1. cd ~/yocto-docs/documentation 2. git grep "&DISTRO;" * This search reveals all the occurrences of that variable in the entire documentation set. I then went through the affected manuals for that particular ENTITY variable and noted whether they expanded or not. I am including two files to show the results: grep-results and entity-errors. The grep-results shows each occurrence of the &DISTRO; variable in all the XML source files. The entity-errors file is my listing of each found result followed by what actually appears in the HTML version of the manual. Some work and some don't. I am also including the ~/yocto-docs/documentation/poky.ent file as well. It is possible to work-around this by not using the poky.ent file and simply hand-supplying the proper strings. This greatly increases the steps to produce each release of the documentation, however, and introduces the possibility of errors. This is likely something simple but I have spent too much time on it already so I am hoping someone can help. Scott
Created attachment 2319 [details] Explanation for &DISTRO; expansions (WORKED and FAILED)
Created attachment 2320 [details] poky.ent file with all the ENTITY declarations.
adding self to CC list and others
I can confirm I can reproduce this on Ubuntu 14.10 myself. I tried building libxslt 1.1.26, the version in Ubuntu 12.04 and using that, the same problem exists there. its therefore likely something in one of the dependencies of libxslt such as libxml2 which is causing the issue.
Additionally, reverting to libxml 2.7.8 fixes the generated document so it looks like a libxml2 issue.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=93739f10bb026a7ec8f1a3d21c309980471c36bd and the preceeding commit have my debug info so far
The issue is caused by the CVE-2014-3660 fix applied to libxml2. Its fixed in master of libxml2 but not in any release. The fix is: https://git.gnome.org/browse/libxml2/commit/?id=72a46a519ce7326d9a00f0b6a7f2a8e958cd1675 We've seen such differences in testing as some people have fixed CVE, some have not. We should integrate the CVE into Yocto, then give Scott instructions on how to use the Yocto built xsltproc command until the fix makes it into the various distros.
Just to capture the commit log for history: Add to local.conf: PREFERRED_VERSION_libxslt-native = "1.1.26" PREFERRED_VERSION_libxml2-native = "2.7.8" Reproducer for doc bug: cd documentation/adt-manual /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml With bug: grep /opt/ * -r | grep env adt-command.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux adt-manual.html: /opt/poky//environment-setup-x86_64-poky-linux adt-manual.html: $ source /opt/poky//environment-setup-i586-poky-linux adt-manual.html: $ source /opt/poky/1.8/environment-setup-i586-poky-linux adt-prepare.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux (note missing 1.8) without bug: $ grep /opt/ * -r | grep env adt-command.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux adt-manual.html: /opt/poky/1.8/environment-setup-x86_64-poky-linux adt-manual.html: $ source /opt/poky/1.8/environment-setup-i586-poky-linux adt-manual.html: $ source /opt/poky/1.8/environment-setup-i586-poky-linux adt-prepare.xml: $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux (note 1.8 present)
So to work around this issue, we can use the build system to help us. Setup a checkout of poky somewhere and: bitbake libxslt-native then export PATH=<path to buliddir>/tmp/sysroots/x86_64-linux/usr/bin:$PATH then run doc make commands as usual. If you change to a new shell, you need to run the export PATH command again. You need a branch of poky with the fix in it, currently that is master-next but it will merge to master soon. Once the upstream distros update their packages to include the fix to libxml2, this workaround will no longer be needed.
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=10837473b27aa1708866eb2b1b5701daa984e94e
Finding information about resolution of this issue in the various distributions is distressingly difficult; despite being fixed upstream in October, the fix is only now trickling into some distributions, for others it's just not clear what the plans are for fixing it. FWIW, it looks like OpenSuSE has a fix for this in their package source although I can't tell if it has actually been pushed out to package updates. Debian has a bug open but there aren't any comments from anyone other than the submitter at the moment: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768089 I couldn't find anything related for Fedora or Ubuntu.
Setting the documentation flag to "done" for this bug. Scott