Bug 7148

richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --versionUsing libxml 20902, libxslt 10126 and libexslt 815
xsltproc was compiled against libxml 20902, libxslt 10126 and libexslt 815
libxslt 10126 was compiled against libxml 20902
libexslt 815 was compiled against libxml 20902
richard@dax:/media/build1/poky/documentation/adt-manual$ time /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml 
^[[A^[[A
Note: namesp. cut : stripped namespace before processing           
            Yocto Project Application Developer's Guide
        
Note: namesp. cut : processing stripped document                   
            Yocto Project Application Developer's Guide
        

real	0m33.173s
user	0m0.591s
sys	0m0.030s
richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --versionUsing libxml 20902, libxslt 10127 and libexslt 816
xsltproc was compiled against libxml 20902, libxslt 10127 and libexslt 816
libxslt 10127 was compiled against libxml 20902
libexslt 816 was compiled against libxml 20902

so the issue is between 1.1.27 and 1.1.28

With SRCREV = "73e08bf7c36a9145d38f51d37e66529b873c011a"

real	2m9.349s
user	0m0.721s
sys	0m0.081s


SRCREV = "dc11b6b379a882418093ecc8adf11f6166682e8d"

real	2m22.658s
user	0m0.765s
sys	0m0.067s

SRCREV = "3d7ba8075408e5b87fde820746bcacfd86b553c6"

real	2m25.824s
user	0m0.733s
sys	0m0.080s

SRCREV = "31312989ab25faf81f9aa9e41d17e0f8f4fe21e7"

real	1m59.490s
user	0m0.721s
sys	0m0.098s

After some poking around, its this:

do_install_append_class-native () {
    create_wrapper ${D}/${bindir}/xsltproc XML_CATALOG_FILES=${sysconfdir}/xml/catalog.xml
}

which when added to the recipe causes the extra delays. I haven;t looked into why.

create_wrapper() from utils.bbclass is used to execute a command with extra environmental variables, at do_install() it moves the actual command "xstlproc" to "xstlproc.real", and creates another executable "xstlproc" where the variables are declared and then it calls the actual command.

After performing an strace on both xstlproc and xstlproc.real, there's an interesting result, it looks like when calling xstlproc the file "catalog.xml" can't be found and it is then downloaded from upstream.

Indeed, the file catalog.xml can't be found on the sysroot native directory, when calling xsltproc.real (without the environmental variable), it "finds" the local catalog file on the host machine, this would be resolved by adding

DEPENDS += "xmlto-native"

To the libxslt recipe, although doing so, installs the file on the native sysroot, but even then the problem persists, either copying the local catalog file to the native sysroot or changing the environmental variable to point to it solves the problem, so the problem is within the catalog file.

The "catalog.xml" file refers to external entities, in this case local files "docbook-xml.xml" and "docbook-xsl.xml":

[aehernan@aehernan-DevStation xml]$ head -n 18 catalog.xml | tail -n 4
<delegateSystem systemIdStartString="http://www.oasis-open.org/docbook/xml/" catalog="./docbook-xml.xml"/>
<delegatePublic publicIdStartString="-//OASIS//DTD DocBook CALS Table Model" catalog="./docbook-xml.xml"/>
<delegateURI uriStartString="http://docbook.sourceforge.net/release/xsl/" catalog="./docbook-xsl.xml"/>
<delegateSystem systemIdStartString="http://docbook.sourceforge.net/release/xsl/" catalog="./docbook-xsl.xml"/>

These two files are contained on the recipes:

docbook-xml-dtd4-native
docbook-xsl-stylesheets-native

respectively, installing these two packages solves the problem.


time ~/Yocto/poky/build/qemux86/tmp/sysroots/x86_64-linux/usr/bin/xsltproc.real -xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml
Note: namesp. cut : stripped namespace before processing           
            Yocto Project Application Developer's Guide
        
Note: namesp. cut : processing stripped document                   
            Yocto Project Application Developer's Guide
        

real	0m0.327s
user	0m0.309s
sys	0m0.018s


time ~/Yocto/poky/build/qemux86/tmp/sysroots/x86_64-linux/usr/bin/xsltproc -xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml
Note: namesp. cut : stripped namespace before processing           
            Yocto Project Application Developer's Guide
        
Note: namesp. cut : processing stripped document                   
            Yocto Project Application Developer's Guide
        

real	0m0.351s
user	0m0.325s
sys	0m0.024s

In summary the problem was that the necessary local files were not being installed, these files were contained within the docbook-xml-dtd4-native and
docbook-xsl-stylesheets-native respectively, these two are listed as RDEPENDS to xmlto-native, which in theory should have worked, but since natives don't have packages it didn't, adding the following line to the xmlto recipe forces the RDEPENDS of xmlto-native to perform do_populate_sysroot before xmlto'd do_populate_sysroot task, solving the issue:

do_populate_sysroot[rdeptask] = "do_populate_sysroot"

It's good that we've solved this within the build system, but unfortunately Scott is still stuck with this problem - after upgrading to Ubuntu 14.10 it takes him up to 70 minutes to build and publish the full set of docs, which is beyond ridiculous. He already has docbook-xml and docbook-xsl installed, and I can't see any other packages in Ubuntu that look like they would be related.

Obviously we could get him a buildtools-like setup as a workaround, and that might be the quickest fix for the moment, but we do have to think about that subset of our users that will be building our docs themselves and potentially hitting the same issue.

Hi, 

Thanks Richard and Alejandro for finding this work-around.  Thanks Paul for helping me get it going.  I am able to build manuals now at the speeds prior to this issue.  As Paul noted, though, it is still a work-around that forces me to use the poky build system as well as export the PATH variable (see bug 7134) to address the ENTITY expansion problems.  

It would be good to have a solution in place that works without exporting the PATH variable, thus making me use the poky/build area to create the manuals.  Maybe the whole issue is outside the scope of Yocto and has to do with Ubuntu.

Scott

Fixed in oe-core 928d41cb7343c23f586a2a1a5e6f81569d9c261b.

Ross, 

One thing about this bug is that if a user clones the poky repo, they still need to take steps to get decent build times.  They need to re-bake the xmlto-native package and then they need to export the PATH variable.  How are we going to handle this for the user?  Do we need to have some sort of release note that instructs the user to take these steps?

Scott

If you're having slow builds outside of the Yocto environment (i.e. you haven't run oe-init-build-env) then the problem is with the host distribution, not anything in Poky.

Yes, obviously, but I don't necessarily agree that that means we don't need to do anything further, because it will still be a problem for our users. Perhaps we can simply report the issue upstream to the major distributions that haven't already addressed it?