Bug 12535

Assigning to Bruce as he's kernel.

If you have taskstats can you verify that this is do_package() and not do_package_rpm or similar.  Or is it all of them.

I suspect the basic problem is that kernel-devsrc is huge.  It's possible that there's an algorithmic problem in the classes somewhere but its more likely just to be linear scaling on a package an order of magnitude larger than most others.

It is do_package, do_package_qa and do_package_rpm are rather fast.
And yes, we have few hundred MB of sources...

buildstats for me:

ross@flashheart /data/poky-tmp/master/buildstats/20180207125644/kernel-devsrc-1.0-r0
$ grep Elapsed  *
do_install:Elapsed time: 17.05 seconds
do_package:Elapsed time: 218.46 seconds
do_packagedata:Elapsed time: 4.95 seconds
do_package_qa:Elapsed time: 34.58 seconds
do_package_write_ipk:Elapsed time: 252.06 seconds
do_package_write_rpm:Elapsed time: 201.40 seconds
do_populate_lic_setscene:Elapsed time: 0.20 seconds
do_prepare_recipe_sysroot:Elapsed time: 0.14 seconds
do_rm_work:Elapsed time: 0.97 seconds

Slow, but not astronomically.  This was on tmpfs though...

Yah, it has always been slow. Mainly because of the amount of small files being slung around. The same reason why we moved the kernel into work-shared to avoid the time penalty of moving the source around the tree for a build.

That being said, I can definitely poke at this more and see if something sub-optimal has crept into the steps.

(In reply to comment #3)
> buildstats for me:
> 
> ross@flashheart
> /data/poky-tmp/master/buildstats/20180207125644/kernel-devsrc-1.0-r0
> $ grep Elapsed  *
> do_install:Elapsed time: 17.05 seconds
> do_package:Elapsed time: 218.46 seconds
> do_packagedata:Elapsed time: 4.95 seconds
> do_package_qa:Elapsed time: 34.58 seconds
> do_package_write_ipk:Elapsed time: 252.06 seconds
> do_package_write_rpm:Elapsed time: 201.40 seconds
> do_populate_lic_setscene:Elapsed time: 0.20 seconds
> do_prepare_recipe_sysroot:Elapsed time: 0.14 seconds
> do_rm_work:Elapsed time: 0.97 seconds
> 
> Slow, but not astronomically.  This was on tmpfs though...

Well, my numbers are quite a bit worse for do_package:

/data/master/poky-contrib/build2-nuc-sato-master-multilib-distro-1-2018-02-05/tmp/buildstats/20180206002151/kernel-devsrc-1.0-r0$ grep Elapsed *
do_deploy_sde:Elapsed time: 0.27 seconds
do_install:Elapsed time: 155.82 seconds
do_package:Elapsed time: 5110.79 seconds
do_packagedata:Elapsed time: 0.36 seconds
do_package_qa:Elapsed time: 25.88 seconds
do_package_write_deb:Elapsed time: 27.16 seconds
do_package_write_ipk:Elapsed time: 368.88 seconds
do_package_write_rpm:Elapsed time: 349.06 seconds
do_populate_lic:Elapsed time: 1.26 seconds
do_prepare_recipe_sysroot:Elapsed time: 8.71 seconds

Having said that, I do build with some extra patches, so I'll double check with another build without those.

It is probably worthwhile mentioning at the time the package was being built, I was deleting some older build folders (running multiple rm -rf <old-builddir> & )

When repeating the build in ramdisk, there were no problems. I believe the exceedingly long time for do_package is caused by rm" running in the background, and there is probably not much that can be done about it (short of maybe using different file system).

Juro, You could try:
$ ionice -c 3 rm -rf old-build
to see if that interferes less with the primary build when both are using a disk-backed filesystem. If you don't use ionice then the filesystem *should* treat both the build and the clean-up as equal priority jobs as you probably know.

That said, a slowdown from ~3-500 seconds to ~5000 seconds seems wrong and might be worth trying to understand by doing additional systematic tests.

(In reply to comment #8)
> Juro, You could try:
> $ ionice -c 3 rm -rf old-build
> to see if that interferes less with the primary build when both are using a
> disk-backed filesystem. If you don't use ionice then the filesystem *should*
> treat both the build and the clean-up as equal priority jobs as you probably
> know.
> 
> That said, a slowdown from ~3-500 seconds to ~5000 seconds seems wrong and
> might be worth trying to understand by doing additional systematic tests.

Thanks, I was not using ionice, so it will be interesting to see how it improves the do_package performance. I am constantly running out of disk space, so I keep deleting old builds frequently. Therefore I was getting those "unreasonable" times for do_package of krenel-devsrc frequently as well, failing to connect the dots.

I plan on running some benchmarks (obtained by profiling via bitbake -P) and will post the results (with/without rm and/or ionice).

I have a significantly streamlined kernel-devsrc package under test.

I'm just wondering where I can find the set of tests for it .. since I'm sure, I've dropped some required elements :D

(In reply to comment #10)
> I have a significantly streamlined kernel-devsrc package under test.
> 
> I'm just wondering where I can find the set of tests for it .. since I'm
> sure, I've dropped some required elements :D

I generally do

local.conf : INHERIT += "testimage"
$ MACHINE=xxx bitbake core-image-sato-sdk
$ MACHINE=xxx bitbake core-image-sato-sdk -c testimage

(I would do this for several machines)

One of the tests is building external kernel module. 
Takes a while.

I was able to build and launch, but of course my runqemu failed due to an SDK issue. On my servers, I always run "nographic".

Does anyone have a pointer where the testimage stuff is documented ? I looked, but can't find the tests, how to run a single test, or how to modify the qemu parameters to have nographic in play.

I found the tests/and suites variables via the bbclass. still searching on how to pass 'nographic' to the run, perhaps I need a non-sato based image. ..

following the steps in the qa test, I scp'd the components onto my target and ran the test manually:

root@qemux86-64:/tmp# make
make -C /lib/modules/4.14.16-yocto-standard/build/ M=/tmp modules
make[1]: Entering directory '/lib/modules/4.14.16-yocto-standard/build'
  CC [M]  /tmp/hellomod.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/hellomod.mod.o
  LD [M]  /tmp/hellomod.ko
make[1]: Leaving directory '/lib/modules/4.14.16-yocto-standard/build'
root@qemux86-64:/tmp# insmod hellomod.ko   
[  331.162855] hellomod: loading out-of-tree module taints kernel.
[  331.167231] Hello world!

------

I made one change, in that the kernel build infrastructure is normalized to what you find in other distros: /lib/modules/<version>/build/...

This is all done with a kerneldev package that is:

-rw-r--r-- 2 bruce bruce 9.4M Feb 14 18:13 kernel-devsrc-1.0-r0.qemux86_64.rpm

So 9.4Megs, shouldn't break the bank in terms of i/o or in image size.

9.4M down from 600M? I owe you a beer.

(In reply to comment #15)
> 9.4M down from 600M? I owe you a beer.

(In reply to comment #14)
> following the steps in the qa test, I scp'd the components onto my target
> and ran the test manually:
> 
> root@qemux86-64:/tmp# make
> make -C /lib/modules/4.14.16-yocto-standard/build/ M=/tmp modules
> make[1]: Entering directory '/lib/modules/4.14.16-yocto-standard/build'
>   CC [M]  /tmp/hellomod.o
>   Building modules, stage 2.
>   MODPOST 1 modules
>   CC      /tmp/hellomod.mod.o
>   LD [M]  /tmp/hellomod.ko
> make[1]: Leaving directory '/lib/modules/4.14.16-yocto-standard/build'
> root@qemux86-64:/tmp# insmod hellomod.ko   
> [  331.162855] hellomod: loading out-of-tree module taints kernel.
> [  331.167231] Hello world!
> 
> ------
> 
> I made one change, in that the kernel build infrastructure is normalized to
> what you find in other distros: /lib/modules/<version>/build/...
> 
> This is all done with a kerneldev package that is:
> 
> -rw-r--r-- 2 bruce bruce 9.4M Feb 14 18:13
> kernel-devsrc-1.0-r0.qemux86_64.rpm
> 
> So 9.4Megs, shouldn't break the bank in terms of i/o or in image size.

Yes, the link is quite useful. I had to put it manually in BuildAppliance recipe:
lnr ${IMAGE_ROOTFS}${KERNEL_SRC_PATH} ${IMAGE_ROOTFS}/lib/modules/${KERNEL_VERSION}/build
in order to be able to build VirtualBox guest extensions (as an out-of-tree module). Having said that, I will test the new kernel-devsrc package with BuildAppliance as well...(w/o the link)

Once this is merged, I will resolve
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6630
and
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4389

The re-worked kernel-devsrc package addresses this issue