15568 2024-07-31 22:18:24 +0000 Improve disk space checks on workers 2024-08-05 11:57:55 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple RESOLVED FIXED Medium+ normal 5.1 M3 1 alexandre.belloni richard.purdie infras.ab.watcher Infras.watcher pidge --- --- No (bug/feature does not impact docs) oldest_to_newest 99481 0 alexandre.belloni 2024-07-31 22:18:24 +0000 Low disk space on a worker can have a catastrophic effect on builds. For example, this a-quick had plenty of failures because one worker had low space on disk: https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2526 What happens is that the build fails but as the worker is still available, it will be selected to run the next build which fails for the same reason and so n... Disk space checks need to be improved to avoid this situation. 99482 1 richard.purdie 2024-08-01 08:37:55 +0000 I suspect we only monitor the space on one disk and we need to monitor it on all key partitions 99489 2 richard.purdie 2024-08-04 11:38:40 +0000 Disk space code is here: https://git.yoctoproject.org/yocto-autobuilder2/tree/builders.py#n68 99490 3 richard.purdie 2024-08-04 12:01:26 +0000 Looking at a recently failure, it is the main disk filling up: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/7021/steps/14/logs/warnings although: https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2526/steps/13/logs/warnings shows it was /tmp that filled up. I've pushed https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=d809e29a23d5614d8873c3f5f1fb7293da96cdd0 which should help and make it easier to add other mount points. 99492 4 richard.purdie 2024-08-05 11:57:55 +0000 I opened https://bugzilla.yoctoproject.org/show_bug.cgi?id=15569 for further work, closing this one as the multiple dirs issue has been fixed