Per the process on https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan Ubuntu 17.04 should be a supported distro.
(In reply to comment #0) > Per the process on https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan > Ubuntu 17.04 should be a supported distro. Ubuntu17.04 past all test-matrix except nightly-world, you said that this issue is because the host is running out of memory and OOMing the webkit build, but the host has 32Gb of RAM. Rev:eff56e4f0d59b1d965a68e4f009b7f07717b7edd Errors on ubuntu 17.04, nightly-world : ERROR: webkitgtk-2.14.5-r0 do_compile: oe_runmake failed ERROR: webkitgtk-2.14.5-r0 do_compile: Function failed: do_compile (log file is located at /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky-linux/webkitgtk/2.14.5-r0/temp/l og.do_compile.18091) ERROR: Logfile of failure stored in: /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky-linux/webkitgtk/2.14.5-r0/temp/log.do_compile.18091
(In reply to comment #1) > (In reply to comment #0) > > Per the process on https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan > > Ubuntu 17.04 should be a supported distro. > > > Ubuntu17.04 past all test-matrix except nightly-world, you said that this > issue is because the host is running out of memory and OOMing the webkit > build, but the host has 32Gb of RAM. > > Rev:eff56e4f0d59b1d965a68e4f009b7f07717b7edd > > Errors on ubuntu 17.04, nightly-world : > ERROR: webkitgtk-2.14.5-r0 do_compile: oe_runmake failed > ERROR: webkitgtk-2.14.5-r0 do_compile: Function failed: do_compile (log file > is located at > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky- > linux/webkitgtk/2.14.5-r0/temp/l og.do_compile.18091) > ERROR: Logfile of failure stored in: > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky- > linux/webkitgtk/2.14.5-r0/temp/log.do_compile.18091 Apologies, I'd completely forgotten how this had been left. Master has changed a lot since that build was ran, including an upgrade of webkit to 2.16.1 — any chance you could run the Ubuntu 17.04 build again and see where we are? In case of failures please attach logs here.
Created attachment 3800 [details] nightly-world-log
(In reply to comment #2) > (In reply to comment #1) > > (In reply to comment #0) > > > Per the process on https://wiki.yoctoproject.org/wiki/Distro_Testing_Plan > > > Ubuntu 17.04 should be a supported distro. > > > > > > Ubuntu17.04 past all test-matrix except nightly-world, you said that this > > issue is because the host is running out of memory and OOMing the webkit > > build, but the host has 32Gb of RAM. > > > > Rev:eff56e4f0d59b1d965a68e4f009b7f07717b7edd > > > > Errors on ubuntu 17.04, nightly-world : > > ERROR: webkitgtk-2.14.5-r0 do_compile: oe_runmake failed > > ERROR: webkitgtk-2.14.5-r0 do_compile: Function failed: do_compile (log file > > is located at > > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky- > > linux/webkitgtk/2.14.5-r0/temp/l og.do_compile.18091) > > ERROR: Logfile of failure stored in: > > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky- > > linux/webkitgtk/2.14.5-r0/temp/log.do_compile.18091 > > Apologies, I'd completely forgotten how this had been left. Master has > changed a lot since that build was ran, including an upgrade of webkit to > 2.16.1 — any chance you could run the Ubuntu 17.04 build again and see where > we are? > > In case of failures please attach logs here. the webkit has been updated to 2.16.6 version but there are another errors building world image.. Please refer to attachment.
Could you also share the logs of the failed tasks? /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/x86-pokymllib32-linux/lib32-webkitgtk/2.16.1-r0/temp/log.do_compile.15439 /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky-linux/webkitgtk/2.16.1-r0/temp/log.do_compile.16052 /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/x86-pokymllib32-linux/lib32-piglit/1.0+gitrAUTOINC+ca58eec0b9-r0/temp/log.do_compile.32256 /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky-linux/piglit/1.0+gitrAUTOINC+ca58eec0b9-r0/temp/log.do_compile.26271
Created attachment 3801 [details] log.do_compile.15439
Created attachment 3802 [details] log.do_compile.16052
Created attachment 3803 [details] log.do_compile.26271
Created attachment 3804 [details] log.do_compile.32256
(In reply to comment #5) > Could you also share the logs of the failed tasks? > > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/x86- > pokymllib32-linux/lib32-webkitgtk/2.16.1-r0/temp/log.do_compile.15439 > > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky- > linux/webkitgtk/2.16.1-r0/temp/log.do_compile.16052 > > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/x86- > pokymllib32-linux/lib32-piglit/1.0+gitrAUTOINC+ca58eec0b9-r0/temp/log. > do_compile.32256 > > /yocto-autobuilder/yocto-worker/nightly-world/build/build/tmp/work/i586-poky- > linux/piglit/1.0+gitrAUTOINC+ca58eec0b9-r0/temp/log.do_compile.26271 done!
(In reply to comment #1) > Ubuntu17.04 past all test-matrix except nightly-world, you said that this > issue is because the host is running out of memory and OOMing the webkit > build, but the host has 32Gb of RAM. When you say "host" does this refer to memory on the machine that is doing the build or does the build happen in a virtual machine?
(In reply to comment #11) > (In reply to comment #1) > > Ubuntu17.04 past all test-matrix except nightly-world, you said that this > > issue is because the host is running out of memory and OOMing the webkit > > build, but the host has 32Gb of RAM. > > When you say "host" does this refer to memory on the machine that is doing > the build or does the build happen in a virtual machine? It is referred to memory on the virtual machine.
So I've done some more careful tests (on Debian stable so not related to Ubuntu as such). I defined a highly parallel setup ("-j 44") and have 32GB of actual memory, with less than 1GB used before starting the compile. I'm estimating memory usage by making sure bitbake only runs the specific compile task (and not much else is running on machine) and running while true; do free -m | grep buffers/cache >> mem.log; sleep 1; done to log things. "Used" column for piglit compile task alone peaks at 4GB. "Used" column for webkitgtk compile task alone peaks at 30GB. My conclusion is that we cannot expect to be able to do a massively parallel compile of even webkit-gtk alone with 32 GB, let alone with other tasks going on at the same time. Ideas: 1. swap: Maybe ensure the machine has at least 32 GB of swap? I get that using swap on a build server would be suboptimal but this would be just for those short peaks during webkit compile 2. Modify webkit recipe so we don't do massive PARALLEL_MAKE. I'm not yet certain how much this lowers the peaks (just a couple of linker instances might be responsible for most of them): I will update when I have some more test results.
(In reply to comment #13) > 2. Modify webkit recipe so we don't do massive PARALLEL_MAKE. I'm not yet > certain how much this lowers the peaks (just a couple of linker instances > might be responsible for most of them): I will update when I have some more > test results. This does help: webkit-gtk compile with 44 threads : peak used memory 30GB webkit-gtk compile with 8 threads : peak used memory 7GB Unfortunately it also makes the build a lot slower (webkit parallelizes really well). My #1 suggestion is to make sure the machine has enough swap.
The machines in the public autobuilder clusters have 4GB of swap.
(In reply to comment #14) > (In reply to comment #13) > > 2. Modify webkit recipe so we don't do massive PARALLEL_MAKE. I'm not yet > > certain how much this lowers the peaks (just a couple of linker instances > > might be responsible for most of them): I will update when I have some more > > test results. > > This does help: > webkit-gtk compile with 44 threads : peak used memory 30GB > webkit-gtk compile with 8 threads : peak used memory 7GB > Unfortunately it also makes the build a lot slower (webkit parallelizes > really well). > > > My #1 suggestion is to make sure the machine has enough swap. I increased swap memory as your recommendation and nightly-world run successfully.
(In reply to comment #15) > The machines in the public autobuilder clusters have 4GB of swap. Joshua, In order to add the new distro on GDC/AB.. Do you know if ubuntu1604 will be replaced for ubuntu1704? or Do we need to have both distributions?.
We'd want to keep Ubuntu 16.04 as that's the most recent LTS. We should replace the other Ubuntu worker (16.10) with 17.04
You absolutely need lots of swap. I'm running a webkit build now and I see: total used free shared buff/cache available Mem: 32781964 31313328 970036 1324 498600 1076244 Swap: 18989052 5352032 13637020 I will resist any attempts to make webkit build even slower than it is now ;-)
(In reply to comment #19) > You absolutely need lots of swap. I'm running a webkit build now and I see: > > total used free shared buff/cache > available > Mem: 32781964 31313328 970036 1324 498600 > 1076244 > Swap: 18989052 5352032 13637020 > > I will resist any attempts to make webkit build even slower than it is now > ;-) Thank you for your findings. We will increase swap not only on ubuntu1704 but also on all our GDC/AB workers.
Ubuntu17.04 has been added on GDC/AB as supported distro