Bug 12310

Summary:	Workers running out of space causing STOPTASKS to trigger
Product:	[Infrastructure] AutoBuilder	Reporter:	Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Component:	autobuilder	Assignee:	Michael Halstead <mhalstead>
Status:	RESOLVED DUPLICATE	QA Contact:
Severity:	critical
Priority:	Undecided	CC:	infras.ab.watcher, Infras.watcher, joshuagloe, mhalstead, pidge, ross.burton
Version:	2.5
Target Milestone:	---
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Leonardo Sandoval Gonzalez 2017-11-06 16:42:24 UTC

Apparently, AB worker needs some more space to work properly:

-----
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
------

AB build info:

-----

Build status: FAILURE Build worker for this Build: centos7.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-x86-lsb/builds/563
Build Reason: Triggerable(trigger_main-build)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS &#43;= " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta-yocto-bsp \
"

• BuildImages : core-image-lsb core-image-lsb-sdk
• DEST : /srv/autobuilder/autobuilder.yoctoproject.org/pub/nightly/20171105-2
• DISTRO : poky-lsb
• KEEP_BUILD_HIST : False
• Contents of auto.conf (local.conf for autobuilders)
INHERIT &#43;= "report-error"
DISTRO = "poky-lsb"
PACKAGE_CLASSES = "package_rpm package_deb package_ipk"
BB_NUMBER_THREADS = "24"
PARALLEL_MAKE = "-j 28"
SDKMACHINE ?= "i686"
BB_TASK_NICE_LEVEL = "5"
BB_TASK_NICE_LEVEL_task-testimage = "0"
BB_TASK_IONICE_LEVEL = "2.7"
BB_TASK_IONICE_LEVEL_task-testimage = "2.1"
DL_DIR = "/srv/autobuilder/autobuilder.yoctoproject.org/current_sources"
SSTATE_DIR ?= "/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstate/"
MACHINE = "genericx86"
PREMIRRORS = ""
BB_GENERATE_MIRROR_TARBALLS = "1"
PRSERV_HOST = "localhost:0"
ADTREPO = "http://adtrepo.yoctoproject.org//2.4"
QEMU_USE_KVM = "True"
SDK_EXT_TYPE = "minimal"
SDK_INCLUDE_TOOLCHAIN = "1"
SANITY_TESTED_DISTROS = ""
----

Comment 1 Leonardo Sandoval Gonzalez 2017-11-06 16:43:31 UTC

Another affected AB build:

https://autobuilder.yocto.io/builders/nightly-packagemanagers/builds/207

Comment 2 Leonardo Sandoval Gonzalez 2017-11-06 16:44:27 UTC

This is happening on another worker:

Build status: FAILURE Build worker for this Build: fedora26.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-world/builds/545
Build Reason: Triggerable(trigger_main-build_nowait)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS &#43;= " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta-yocto-bsp \
"

Comment 3 Leonardo Sandoval Gonzalez 2017-11-06 16:46:31 UTC

and opensuse423:

Build status: FAILURE Build worker for this Build: opensuse423.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-x86-64/builds/571
Build Reason: Triggerable(trigger_main-build)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS &#43;= " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta-yocto-bsp \
"

Comment 4 Leonardo Sandoval Gonzalez 2017-11-06 16:47:59 UTC

and ubuntu 1604.

In other words, host OS, alias the builders, needs more space and/or cleanup past builds.


Build status: FAILURE Build worker for this Build: ubuntu1604.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-musl-x86-64/builds/24
Build Reason: Triggerable(trigger_main-build_nowait)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS &#43;= " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta-yocto-bsp \
"

Comment 5 Joshua Lock 2017-11-06 16:55:59 UTC

Workers aren't running out of space, they think there aren't enough free inodes. Richard has done some good investigation and Michael is looking into this.

*** This bug has been marked as a duplicate of bug 12267 ***