12310 2017-11-06 16:42:24 +0000 Workers running out of space causing STOPTASKS to trigger 2017-11-06 16:55:59 +0000 1 1 5 Infrastructure AutoBuilder autobuilder 2.5 x86 Multiple RESOLVED DUPLICATE 12267 Undecided critical --- 1 leonardo.sandoval.gonzalez mhalstead infras.ab.watcher Infras.watcher joshuagloe mhalstead pidge ross.burton --- --- No (bug/feature does not impact docs) oldest_to_newest 77987 0 leonardo.sandoval.gonzalez 2017-11-06 16:42:24 +0000 Apparently, AB worker needs some more space to work properly: ----- ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"! ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"! Summary: There were 2 ERROR messages shown, returning a non-zero exit code. ------ AB build info: ----- Build status: FAILURE Build worker for this Build: centos7.yocto.io Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-x86-lsb/builds/563 Build Reason: Triggerable(trigger_main-build) Build Properties: • Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly) POKY_BBLAYERS_CONF_VERSION = BBPATH = "${TOPDIR}" BBFILES ?="" BBLAYERS += " \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta-poky \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta-yocto-bsp \ " • BuildImages : core-image-lsb core-image-lsb-sdk • DEST : /srv/autobuilder/autobuilder.yoctoproject.org/pub/nightly/20171105-2 • DISTRO : poky-lsb • KEEP_BUILD_HIST : False • Contents of auto.conf (local.conf for autobuilders) INHERIT += "report-error" DISTRO = "poky-lsb" PACKAGE_CLASSES = "package_rpm package_deb package_ipk" BB_NUMBER_THREADS = "24" PARALLEL_MAKE = "-j 28" SDKMACHINE ?= "i686" BB_TASK_NICE_LEVEL = "5" BB_TASK_NICE_LEVEL_task-testimage = "0" BB_TASK_IONICE_LEVEL = "2.7" BB_TASK_IONICE_LEVEL_task-testimage = "2.1" DL_DIR = "/srv/autobuilder/autobuilder.yoctoproject.org/current_sources" SSTATE_DIR ?= "/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstate/" MACHINE = "genericx86" PREMIRRORS = "" BB_GENERATE_MIRROR_TARBALLS = "1" PRSERV_HOST = "localhost:0" ADTREPO = "http://adtrepo.yoctoproject.org//2.4" QEMU_USE_KVM = "True" SDK_EXT_TYPE = "minimal" SDK_INCLUDE_TOOLCHAIN = "1" SANITY_TESTED_DISTROS = "" ---- 77988 1 leonardo.sandoval.gonzalez 2017-11-06 16:43:31 +0000 Another affected AB build: https://autobuilder.yocto.io/builders/nightly-packagemanagers/builds/207 77989 2 leonardo.sandoval.gonzalez 2017-11-06 16:44:27 +0000 This is happening on another worker: Build status: FAILURE Build worker for this Build: fedora26.yocto.io Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-world/builds/545 Build Reason: Triggerable(trigger_main-build_nowait) Build Properties: • Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly) POKY_BBLAYERS_CONF_VERSION = BBPATH = "${TOPDIR}" BBFILES ?="" BBLAYERS += " \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta-poky \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta-yocto-bsp \ " 77990 3 leonardo.sandoval.gonzalez 2017-11-06 16:46:31 +0000 and opensuse423: Build status: FAILURE Build worker for this Build: opensuse423.yocto.io Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-x86-64/builds/571 Build Reason: Triggerable(trigger_main-build) Build Properties: • Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly) POKY_BBLAYERS_CONF_VERSION = BBPATH = "${TOPDIR}" BBFILES ?="" BBLAYERS += " \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta-poky \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta-yocto-bsp \ " 77991 4 leonardo.sandoval.gonzalez 2017-11-06 16:47:59 +0000 and ubuntu 1604. In other words, host OS, alias the builders, needs more space and/or cleanup past builds. Build status: FAILURE Build worker for this Build: ubuntu1604.yocto.io Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-musl-x86-64/builds/24 Build Reason: Triggerable(trigger_main-build_nowait) Build Properties: • Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly) POKY_BBLAYERS_CONF_VERSION = BBPATH = "${TOPDIR}" BBFILES ?="" BBLAYERS += " \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta-poky \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta-yocto-bsp \ " 77992 5 joshuagloe 2017-11-06 16:55:59 +0000 Workers aren't running out of space, they think there aren't enough free inodes. Richard has done some good investigation and Michael is looking into this. *** This bug has been marked as a duplicate of bug 12267 ***