Bug 12310 - Workers running out of space causing STOPTASKS to trigger
Summary: Workers running out of space causing STOPTASKS to trigger
Status: RESOLVED DUPLICATE of bug 12267
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: 2.5
Hardware: x86 Multiple
: Undecided critical
Target Milestone: ---
Assignee: Michael Halstead
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-11-06 16:42 UTC by Leonardo Sandoval Gonzalez
Modified: 2017-11-06 16:55 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Leonardo Sandoval Gonzalez 2017-11-06 16:42:24 UTC 
Apparently, AB worker needs some more space to work properly:

-----
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
------

AB build info:

-----

Build status: FAILURE Build worker for this Build: centos7.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-x86-lsb/builds/563
Build Reason: Triggerable(trigger_main-build)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS += " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-lsb/build/meta-yocto-bsp \
"

• BuildImages : core-image-lsb core-image-lsb-sdk
• DEST : /srv/autobuilder/autobuilder.yoctoproject.org/pub/nightly/20171105-2
• DISTRO : poky-lsb
• KEEP_BUILD_HIST : False
• Contents of auto.conf (local.conf for autobuilders)
INHERIT += "report-error"
DISTRO = "poky-lsb"
PACKAGE_CLASSES = "package_rpm package_deb package_ipk"
BB_NUMBER_THREADS = "24"
PARALLEL_MAKE = "-j 28"
SDKMACHINE ?= "i686"
BB_TASK_NICE_LEVEL = "5"
BB_TASK_NICE_LEVEL_task-testimage = "0"
BB_TASK_IONICE_LEVEL = "2.7"
BB_TASK_IONICE_LEVEL_task-testimage = "2.1"
DL_DIR = "/srv/autobuilder/autobuilder.yoctoproject.org/current_sources"
SSTATE_DIR ?= "/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstate/"
MACHINE = "genericx86"
PREMIRRORS = ""
BB_GENERATE_MIRROR_TARBALLS = "1"
PRSERV_HOST = "localhost:0"
ADTREPO = "http://adtrepo.yoctoproject.org//2.4"
QEMU_USE_KVM = "True"
SDK_EXT_TYPE = "minimal"
SDK_INCLUDE_TOOLCHAIN = "1"
SANITY_TESTED_DISTROS = ""
----
Comment 1 Leonardo Sandoval Gonzalez 2017-11-06 16:43:31 UTC 
Another affected AB build:

https://autobuilder.yocto.io/builders/nightly-packagemanagers/builds/207
Comment 2 Leonardo Sandoval Gonzalez 2017-11-06 16:44:27 UTC 
This is happening on another worker:

Build status: FAILURE Build worker for this Build: fedora26.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-world/builds/545
Build Reason: Triggerable(trigger_main-build_nowait)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS += " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-world/build/meta-yocto-bsp \
"
Comment 3 Leonardo Sandoval Gonzalez 2017-11-06 16:46:31 UTC 
and opensuse423:

Build status: FAILURE Build worker for this Build: opensuse423.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-x86-64/builds/571
Build Reason: Triggerable(trigger_main-build)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS += " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/meta-yocto-bsp \
"
Comment 4 Leonardo Sandoval Gonzalez 2017-11-06 16:47:59 UTC 
and ubuntu 1604.

In other words, host OS, alias the builders, needs more space and/or cleanup past builds.


Build status: FAILURE Build worker for this Build: ubuntu1604.yocto.io
Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-musl-x86-64/builds/24
Build Reason: Triggerable(trigger_main-build_nowait)
Build Properties:
• Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly)
POKY_BBLAYERS_CONF_VERSION =
BBPATH = "${TOPDIR}"
BBFILES ?=""
BBLAYERS += " \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta-poky \
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-musl-x86-64/build/meta-yocto-bsp \
"
Comment 5 Joshua Lock 2017-11-06 16:55:59 UTC 
Workers aren't running out of space, they think there aren't enough free inodes. Richard has done some good investigation and Michael is looking into this.

*** This bug has been marked as a duplicate of bug 12267 ***