7148 2015-01-15 21:21:48 +0000 Building Yocto Docs takes a significant time with recent tools versions 2015-02-12 18:14:01 +0000 1 1 9 Documentation General Docs docs-general unspecified x86 Multiple RESOLVED FIXED 28 January 2015: work-around verified Medium normal 1.8 1 richard.purdie alejandro.hernandez bluelightning ross.burton sgw srifenbark --- --- No (bug/feature does not impact docs) oldest_to_newest 47951 0 richard.purdie 2015-01-15 21:21:48 +0000 richard@dax:/media/build1/poky/documentation/adt-manual$ time /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml Note: namesp. cut : stripped namespace before processing Yocto Project Application Developer's Guide WARNING: cannot add @xml:base to node set root element. Relative paths may not work. Note: namesp. cut : processing stripped document Yocto Project Application Developer's Guide real 0m32.690s user 0m0.573s sys 0m0.043s richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --version Using libxml 20708, libxslt 10126 and libexslt 815 xsltproc was compiled against libxml 20708, libxslt 10126 and libexslt 815 libxslt 10126 was compiled against libxml 20708 libexslt 815 was compiled against libxml 20708 richard@dax:/media/build1/poky/documentation/adt-manual$ time /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml Note: namesp. cut : stripped namespace before processing Yocto Project Application Developer's Guide Note: namesp. cut : processing stripped document Yocto Project Application Developer's Guide real 2m41.852s user 0m0.720s sys 0m0.099s richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --versionUsing libxml 20902, libxslt 10128 and libexslt 817 xsltproc was compiled against libxml 20902, libxslt 10128 and libexslt 817 libxslt 10128 was compiled against libxml 20902 libexslt 817 was compiled against libxml 20902 richard@dax:/media/build1/poky/documentation/adt-manual$ time /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml Note: namesp. cut : stripped namespace before processing Yocto Project Application Developer's Guide WARNING: cannot add @xml:base to node set root element. Relative paths may not work. Note: namesp. cut : processing stripped document Yocto Project Application Developer's Guide real 0m33.907s user 0m0.563s sys 0m0.041s richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --versionUsing libxml 20902, libxslt 10126 and libexslt 815 xsltproc was compiled against libxml 20902, libxslt 10126 and libexslt 815 libxslt 10126 was compiled against libxml 20902 libexslt 815 was compiled against libxml 20902 so the issue looks to be in libxslt between 1.1.26 and 1.1.28 47952 1 richard.purdie 2015-01-15 21:26:12 +0000 richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --versionUsing libxml 20902, libxslt 10126 and libexslt 815 xsltproc was compiled against libxml 20902, libxslt 10126 and libexslt 815 libxslt 10126 was compiled against libxml 20902 libexslt 815 was compiled against libxml 20902 richard@dax:/media/build1/poky/documentation/adt-manual$ time /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml ^[[A^[[A Note: namesp. cut : stripped namespace before processing Yocto Project Application Developer's Guide Note: namesp. cut : processing stripped document Yocto Project Application Developer's Guide real 0m33.173s user 0m0.591s sys 0m0.030s richard@dax:/media/build1/poky/documentation/adt-manual$ /media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/xsltproc --versionUsing libxml 20902, libxslt 10127 and libexslt 816 xsltproc was compiled against libxml 20902, libxslt 10127 and libexslt 816 libxslt 10127 was compiled against libxml 20902 libexslt 816 was compiled against libxml 20902 so the issue is between 1.1.27 and 1.1.28 47953 2 richard.purdie 2015-01-15 22:05:59 +0000 With SRCREV = "73e08bf7c36a9145d38f51d37e66529b873c011a" real 2m9.349s user 0m0.721s sys 0m0.081s SRCREV = "dc11b6b379a882418093ecc8adf11f6166682e8d" real 2m22.658s user 0m0.765s sys 0m0.067s SRCREV = "3d7ba8075408e5b87fde820746bcacfd86b553c6" real 2m25.824s user 0m0.733s sys 0m0.080s SRCREV = "31312989ab25faf81f9aa9e41d17e0f8f4fe21e7" real 1m59.490s user 0m0.721s sys 0m0.098s 47954 3 richard.purdie 2015-01-15 22:16:23 +0000 After some poking around, its this: do_install_append_class-native () { create_wrapper ${D}/${bindir}/xsltproc XML_CATALOG_FILES=${sysconfdir}/xml/catalog.xml } which when added to the recipe causes the extra delays. I haven;t looked into why. 48011 4 alejandro.hernandez 2015-01-16 21:51:59 +0000 create_wrapper() from utils.bbclass is used to execute a command with extra environmental variables, at do_install() it moves the actual command "xstlproc" to "xstlproc.real", and creates another executable "xstlproc" where the variables are declared and then it calls the actual command. After performing an strace on both xstlproc and xstlproc.real, there's an interesting result, it looks like when calling xstlproc the file "catalog.xml" can't be found and it is then downloaded from upstream. 48014 5 alejandro.hernandez 2015-01-16 23:29:04 +0000 Indeed, the file catalog.xml can't be found on the sysroot native directory, when calling xsltproc.real (without the environmental variable), it "finds" the local catalog file on the host machine, this would be resolved by adding DEPENDS += "xmlto-native" To the libxslt recipe, although doing so, installs the file on the native sysroot, but even then the problem persists, either copying the local catalog file to the native sysroot or changing the environmental variable to point to it solves the problem, so the problem is within the catalog file. 48016 6 alejandro.hernandez 2015-01-17 00:21:03 +0000 The "catalog.xml" file refers to external entities, in this case local files "docbook-xml.xml" and "docbook-xsl.xml": [aehernan@aehernan-DevStation xml]$ head -n 18 catalog.xml | tail -n 4 <delegateSystem systemIdStartString="http://www.oasis-open.org/docbook/xml/" catalog="./docbook-xml.xml"/> <delegatePublic publicIdStartString="-//OASIS//DTD DocBook CALS Table Model" catalog="./docbook-xml.xml"/> <delegateURI uriStartString="http://docbook.sourceforge.net/release/xsl/" catalog="./docbook-xsl.xml"/> <delegateSystem systemIdStartString="http://docbook.sourceforge.net/release/xsl/" catalog="./docbook-xsl.xml"/> These two files are contained on the recipes: docbook-xml-dtd4-native docbook-xsl-stylesheets-native respectively, installing these two packages solves the problem. time ~/Yocto/poky/build/qemux86/tmp/sysroots/x86_64-linux/usr/bin/xsltproc.real -xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml Note: namesp. cut : stripped namespace before processing Yocto Project Application Developer's Guide Note: namesp. cut : processing stripped document Yocto Project Application Developer's Guide real 0m0.327s user 0m0.309s sys 0m0.018s time ~/Yocto/poky/build/qemux86/tmp/sysroots/x86_64-linux/usr/bin/xsltproc -xinclude -o adt-manual.html adt-manual-customization.xsl adt-manual.xml Note: namesp. cut : stripped namespace before processing Yocto Project Application Developer's Guide Note: namesp. cut : processing stripped document Yocto Project Application Developer's Guide real 0m0.351s user 0m0.325s sys 0m0.024s 48342 7 alejandro.hernandez 2015-01-27 21:53:29 +0000 In summary the problem was that the necessary local files were not being installed, these files were contained within the docbook-xml-dtd4-native and docbook-xsl-stylesheets-native respectively, these two are listed as RDEPENDS to xmlto-native, which in theory should have worked, but since natives don't have packages it didn't, adding the following line to the xmlto recipe forces the RDEPENDS of xmlto-native to perform do_populate_sysroot before xmlto'd do_populate_sysroot task, solving the issue: do_populate_sysroot[rdeptask] = "do_populate_sysroot" 48365 8 bluelightning 2015-01-28 15:31:13 +0000 It's good that we've solved this within the build system, but unfortunately Scott is still stuck with this problem - after upgrading to Ubuntu 14.10 it takes him up to 70 minutes to build and publish the full set of docs, which is beyond ridiculous. He already has docbook-xml and docbook-xsl installed, and I can't see any other packages in Ubuntu that look like they would be related. Obviously we could get him a buildtools-like setup as a workaround, and that might be the quickest fix for the moment, but we do have to think about that subset of our users that will be building our docs themselves and potentially hitting the same issue. 48366 9 srifenbark 2015-01-28 16:13:21 +0000 Hi, Thanks Richard and Alejandro for finding this work-around. Thanks Paul for helping me get it going. I am able to build manuals now at the speeds prior to this issue. As Paul noted, though, it is still a work-around that forces me to use the poky build system as well as export the PATH variable (see bug 7134) to address the ENTITY expansion problems. It would be good to have a solution in place that works without exporting the PATH variable, thus making me use the poky/build area to create the manuals. Maybe the whole issue is outside the scope of Yocto and has to do with Ubuntu. Scott 48614 10 ross.burton 2015-02-06 17:24:51 +0000 Fixed in oe-core 928d41cb7343c23f586a2a1a5e6f81569d9c261b. 48774 11 srifenbark 2015-02-12 17:41:44 +0000 Ross, One thing about this bug is that if a user clones the poky repo, they still need to take steps to get decent build times. They need to re-bake the xmlto-native package and then they need to export the PATH variable. How are we going to handle this for the user? Do we need to have some sort of release note that instructs the user to take these steps? Scott 48775 12 ross.burton 2015-02-12 17:46:56 +0000 If you're having slow builds outside of the Yocto environment (i.e. you haven't run oe-init-build-env) then the problem is with the host distribution, not anything in Poky. 48777 13 bluelightning 2015-02-12 18:14:01 +0000 Yes, obviously, but I don't necessarily agree that that means we don't need to do anything further, because it will still be a problem for our users. Perhaps we can simply report the issue upstream to the major distributions that haven't already addressed it?