12535 2018-02-06 21:39:59 +0000 kernel-devsrc do_package takes unreasonably long time 2018-11-30 20:10:35 +0000 1 1 7 Build System, Metadata & Runtime OE-Core kernel 2.5 x86 Multiple RESOLVED FIXED Medium normal 4.99 1 juro.bystricky bruce.ashfield meta.mr.watcher meta.watcher randy.macleod ross.burton tom.zanussi --- --- No (bug/feature does not impact docs) oldest_to_newest 79368 0 juro.bystricky 2018-02-06 21:39:59 +0000 When building an image such as core-image-sato-sdk-ptest, we build kernel-devsrc package as well. I observed it takes over 5501 (!) seconds for kernel-devsrc do_package on my system. Granted, the I/O is stressed quite a bit, but this does not seem reasonable. 79369 1 ross.burton 2018-02-06 22:15:55 +0000 Assigning to Bruce as he's kernel. If you have taskstats can you verify that this is do_package() and not do_package_rpm or similar. Or is it all of them. I suspect the basic problem is that kernel-devsrc is huge. It's possible that there's an algorithmic problem in the classes somewhere but its more likely just to be linear scaling on a package an order of magnitude larger than most others. 79372 2 juro.bystricky 2018-02-06 22:59:02 +0000 It is do_package, do_package_qa and do_package_rpm are rather fast. And yes, we have few hundred MB of sources... 79376 3 ross.burton 2018-02-07 14:04:57 +0000 buildstats for me: ross@flashheart /data/poky-tmp/master/buildstats/20180207125644/kernel-devsrc-1.0-r0 $ grep Elapsed * do_install:Elapsed time: 17.05 seconds do_package:Elapsed time: 218.46 seconds do_packagedata:Elapsed time: 4.95 seconds do_package_qa:Elapsed time: 34.58 seconds do_package_write_ipk:Elapsed time: 252.06 seconds do_package_write_rpm:Elapsed time: 201.40 seconds do_populate_lic_setscene:Elapsed time: 0.20 seconds do_prepare_recipe_sysroot:Elapsed time: 0.14 seconds do_rm_work:Elapsed time: 0.97 seconds Slow, but not astronomically. This was on tmpfs though... 79377 4 bruce.ashfield 2018-02-07 14:08:23 +0000 Yah, it has always been slow. Mainly because of the amount of small files being slung around. The same reason why we moved the kernel into work-shared to avoid the time penalty of moving the source around the tree for a build. That being said, I can definitely poke at this more and see if something sub-optimal has crept into the steps. 79378 5 juro.bystricky 2018-02-07 16:21:24 +0000 (In reply to comment #3) > buildstats for me: > > ross@flashheart > /data/poky-tmp/master/buildstats/20180207125644/kernel-devsrc-1.0-r0 > $ grep Elapsed * > do_install:Elapsed time: 17.05 seconds > do_package:Elapsed time: 218.46 seconds > do_packagedata:Elapsed time: 4.95 seconds > do_package_qa:Elapsed time: 34.58 seconds > do_package_write_ipk:Elapsed time: 252.06 seconds > do_package_write_rpm:Elapsed time: 201.40 seconds > do_populate_lic_setscene:Elapsed time: 0.20 seconds > do_prepare_recipe_sysroot:Elapsed time: 0.14 seconds > do_rm_work:Elapsed time: 0.97 seconds > > Slow, but not astronomically. This was on tmpfs though... Well, my numbers are quite a bit worse for do_package: /data/master/poky-contrib/build2-nuc-sato-master-multilib-distro-1-2018-02-05/tmp/buildstats/20180206002151/kernel-devsrc-1.0-r0$ grep Elapsed * do_deploy_sde:Elapsed time: 0.27 seconds do_install:Elapsed time: 155.82 seconds do_package:Elapsed time: 5110.79 seconds do_packagedata:Elapsed time: 0.36 seconds do_package_qa:Elapsed time: 25.88 seconds do_package_write_deb:Elapsed time: 27.16 seconds do_package_write_ipk:Elapsed time: 368.88 seconds do_package_write_rpm:Elapsed time: 349.06 seconds do_populate_lic:Elapsed time: 1.26 seconds do_prepare_recipe_sysroot:Elapsed time: 8.71 seconds Having said that, I do build with some extra patches, so I'll double check with another build without those. 79396 6 juro.bystricky 2018-02-08 16:13:06 +0000 It is probably worthwhile mentioning at the time the package was being built, I was deleting some older build folders (running multiple rm -rf <old-builddir> & ) 79404 7 juro.bystricky 2018-02-09 16:03:31 +0000 When repeating the build in ramdisk, there were no problems. I believe the exceedingly long time for do_package is caused by rm" running in the background, and there is probably not much that can be done about it (short of maybe using different file system). 79406 8 randy.macleod 2018-02-09 17:06:16 +0000 Juro, You could try: $ ionice -c 3 rm -rf old-build to see if that interferes less with the primary build when both are using a disk-backed filesystem. If you don't use ionice then the filesystem *should* treat both the build and the clean-up as equal priority jobs as you probably know. That said, a slowdown from ~3-500 seconds to ~5000 seconds seems wrong and might be worth trying to understand by doing additional systematic tests. 79409 9 juro.bystricky 2018-02-09 17:13:18 +0000 (In reply to comment #8) > Juro, You could try: > $ ionice -c 3 rm -rf old-build > to see if that interferes less with the primary build when both are using a > disk-backed filesystem. If you don't use ionice then the filesystem *should* > treat both the build and the clean-up as equal priority jobs as you probably > know. > > That said, a slowdown from ~3-500 seconds to ~5000 seconds seems wrong and > might be worth trying to understand by doing additional systematic tests. Thanks, I was not using ionice, so it will be interesting to see how it improves the do_package performance. I am constantly running out of disk space, so I keep deleting old builds frequently. Therefore I was getting those "unreasonable" times for do_package of krenel-devsrc frequently as well, failing to connect the dots. I plan on running some benchmarks (obtained by profiling via bitbake -P) and will post the results (with/without rm and/or ionice). 79441 10 bruce.ashfield 2018-02-14 20:01:03 +0000 I have a significantly streamlined kernel-devsrc package under test. I'm just wondering where I can find the set of tests for it .. since I'm sure, I've dropped some required elements :D 79442 11 juro.bystricky 2018-02-14 20:43:05 +0000 (In reply to comment #10) > I have a significantly streamlined kernel-devsrc package under test. > > I'm just wondering where I can find the set of tests for it .. since I'm > sure, I've dropped some required elements :D I generally do local.conf : INHERIT += "testimage" $ MACHINE=xxx bitbake core-image-sato-sdk $ MACHINE=xxx bitbake core-image-sato-sdk -c testimage (I would do this for several machines) One of the tests is building external kernel module. Takes a while. 79445 12 bruce.ashfield 2018-02-15 04:50:47 +0000 I was able to build and launch, but of course my runqemu failed due to an SDK issue. On my servers, I always run "nographic". Does anyone have a pointer where the testimage stuff is documented ? I looked, but can't find the tests, how to run a single test, or how to modify the qemu parameters to have nographic in play. 79446 13 bruce.ashfield 2018-02-15 05:03:43 +0000 I found the tests/and suites variables via the bbclass. still searching on how to pass 'nographic' to the run, perhaps I need a non-sato based image. .. 79447 14 bruce.ashfield 2018-02-15 05:10:07 +0000 following the steps in the qa test, I scp'd the components onto my target and ran the test manually: root@qemux86-64:/tmp# make make -C /lib/modules/4.14.16-yocto-standard/build/ M=/tmp modules make[1]: Entering directory '/lib/modules/4.14.16-yocto-standard/build' CC [M] /tmp/hellomod.o Building modules, stage 2. MODPOST 1 modules CC /tmp/hellomod.mod.o LD [M] /tmp/hellomod.ko make[1]: Leaving directory '/lib/modules/4.14.16-yocto-standard/build' root@qemux86-64:/tmp# insmod hellomod.ko [ 331.162855] hellomod: loading out-of-tree module taints kernel. [ 331.167231] Hello world! ------ I made one change, in that the kernel build infrastructure is normalized to what you find in other distros: /lib/modules/<version>/build/... This is all done with a kerneldev package that is: -rw-r--r-- 2 bruce bruce 9.4M Feb 14 18:13 kernel-devsrc-1.0-r0.qemux86_64.rpm So 9.4Megs, shouldn't break the bank in terms of i/o or in image size. 79450 15 ross.burton 2018-02-15 10:59:51 +0000 9.4M down from 600M? I owe you a beer. 79451 16 juro.bystricky 2018-02-15 15:05:56 +0000 (In reply to comment #15) > 9.4M down from 600M? I owe you a beer. (In reply to comment #14) > following the steps in the qa test, I scp'd the components onto my target > and ran the test manually: > > root@qemux86-64:/tmp# make > make -C /lib/modules/4.14.16-yocto-standard/build/ M=/tmp modules > make[1]: Entering directory '/lib/modules/4.14.16-yocto-standard/build' > CC [M] /tmp/hellomod.o > Building modules, stage 2. > MODPOST 1 modules > CC /tmp/hellomod.mod.o > LD [M] /tmp/hellomod.ko > make[1]: Leaving directory '/lib/modules/4.14.16-yocto-standard/build' > root@qemux86-64:/tmp# insmod hellomod.ko > [ 331.162855] hellomod: loading out-of-tree module taints kernel. > [ 331.167231] Hello world! > > ------ > > I made one change, in that the kernel build infrastructure is normalized to > what you find in other distros: /lib/modules/<version>/build/... > > This is all done with a kerneldev package that is: > > -rw-r--r-- 2 bruce bruce 9.4M Feb 14 18:13 > kernel-devsrc-1.0-r0.qemux86_64.rpm > > So 9.4Megs, shouldn't break the bank in terms of i/o or in image size. Yes, the link is quite useful. I had to put it manually in BuildAppliance recipe: lnr ${IMAGE_ROOTFS}${KERNEL_SRC_PATH} ${IMAGE_ROOTFS}/lib/modules/${KERNEL_VERSION}/build in order to be able to build VirtualBox guest extensions (as an out-of-tree module). Having said that, I will test the new kernel-devsrc package with BuildAppliance as well...(w/o the link) 79453 17 juro.bystricky 2018-02-15 15:12:45 +0000 Once this is merged, I will resolve https://bugzilla.yoctoproject.org/show_bug.cgi?id=6630 and https://bugzilla.yoctoproject.org/show_bug.cgi?id=4389 82372 18 bruce.ashfield 2018-11-30 20:10:35 +0000 The re-worked kernel-devsrc package addresses this issue