Bug 15568 - Improve disk space checks on workers
Summary: Improve disk space checks on workers
Status: RESOLVED FIXED
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.1 M3
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2024-07-31 22:18 UTC by Alexandre Belloni
Modified: 2024-08-05 11:57 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexandre Belloni 2024-07-31 22:18:24 UTC
Low disk space on a worker can have a catastrophic effect on builds. For example, this a-quick had plenty of failures because one worker had low space on disk:

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2526

What happens is that the build fails but as the worker is still available, it will be selected to run the next build which fails for the same reason and so n...

Disk space checks need to be improved to avoid this situation.
Comment 1 Richard Purdie 2024-08-01 08:37:55 UTC
I suspect we only monitor the space on one disk and we need to monitor it on all key partitions
Comment 2 Richard Purdie 2024-08-04 11:38:40 UTC
Disk space code is here: https://git.yoctoproject.org/yocto-autobuilder2/tree/builders.py#n68
Comment 3 Richard Purdie 2024-08-04 12:01:26 UTC
Looking at a recently failure, it is the main disk filling up:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/7021/steps/14/logs/warnings

although:

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2526/steps/13/logs/warnings

shows it was /tmp that filled up.

I've pushed https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=d809e29a23d5614d8873c3f5f1fb7293da96cdd0 which should help and make it easier to add other mount points.
Comment 4 Richard Purdie 2024-08-05 11:57:55 UTC
I opened https://bugzilla.yoctoproject.org/show_bug.cgi?id=15569 for further work, closing this one as the multiple dirs issue has been fixed