Bug 15568

Summary:	Improve disk space checks on workers
Product:	[Infrastructure] AutoBuilder	Reporter:	Alexandre Belloni <alexandre.belloni>
Component:	autobuilder	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	infras.ab.watcher, Infras.watcher, pidge
Version:	unspecified
Target Milestone:	5.1 M3
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Alexandre Belloni 2024-07-31 22:18:24 UTC

Low disk space on a worker can have a catastrophic effect on builds. For example, this a-quick had plenty of failures because one worker had low space on disk:

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2526

What happens is that the build fails but as the worker is still available, it will be selected to run the next build which fails for the same reason and so n...

Disk space checks need to be improved to avoid this situation.

Comment 1 Richard Purdie 2024-08-01 08:37:55 UTC

I suspect we only monitor the space on one disk and we need to monitor it on all key partitions

Comment 2 Richard Purdie 2024-08-04 11:38:40 UTC

Disk space code is here: https://git.yoctoproject.org/yocto-autobuilder2/tree/builders.py#n68

Comment 3 Richard Purdie 2024-08-04 12:01:26 UTC

Looking at a recently failure, it is the main disk filling up:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/7021/steps/14/logs/warnings

although:

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2526/steps/13/logs/warnings

shows it was /tmp that filled up.

I've pushed https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=d809e29a23d5614d8873c3f5f1fb7293da96cdd0 which should help and make it easier to add other mount points.

Comment 4 Richard Purdie 2024-08-05 11:57:55 UTC

I opened https://bugzilla.yoctoproject.org/show_bug.cgi?id=15569 for further work, closing this one as the multiple dirs issue has been fixed