We recently saw some indexing jobs failing with: Couldn't obtain the lock /srv/autobuilder/valkyrie.yocto.io/pub/non-release/indexing.lock: [Errno 11] Resource temporarily unavailable This is probably because the job before was still running. Looking at all indexing jobs on the autobuilder, we can see some builds are way longer than others. Typical time is about 2 minutes, but some jobs take more than 20 minutes. As an example, builds 52730 and 52731 failed, because of build 52729. 52729 38:33 52730 24:14 52731 9:03
indexing opensuse160-vk-1 master completed at 2026-04-22 11:09:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52730/steps/12/logs/stdio indexing fedora43-vk-2 master completed at 2026-04-22 11:09:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52731/steps/12/logs/stdio indexing debian13-vk-2 master completed at 2026-04-22 17:02:18+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52754/steps/12/logs/stdio
Slow runs may be due to NFS but that's a guess. Can we fix the slow runs?
Looking at https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52729: Generate non-release index: 26m Extract ptest logs: 7m Populate dashboard: 16s The lockfile is obtained and released in the "Generate non-release index" task, which is actually y-a-h/scripts/generate-testresult-index.py. That script does just iterate a lot of files so possibly this is "just" the NFS server struggling.
We do generate lots of logs such as: https://valkyrie.yocto.io/pub/non-release/20260422-99/testresults/qemux86/intermittent_failure_host_data/ and I'm sure more from ptests, etc. Michael, if you collect the FS usage (du -options | zstd > du-log.zst ? ) during maintenance for whatever dir is behind: https://valkyrie.yocto.io/pub/non-release/ I don't want to do it while there are builds running.
We are starting with 719G total in pub/non-release. I can start collecting and logging regularly and add additional detail as well.
We have done some cleanup on the NAS so that there are less files in the directories indexing is traversing, we haven't seen this recur so we can go ahead and close for now.
indexing debian12-vk-4 master completed at 2026-07-09 21:17:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/60107/steps/12/logs/stdio
indexing debian11-vk-1 master completed at 2026-08-10 18:46:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/63106/steps/15/logs/stdio
indexing ubuntu2510-vk-1 master completed at 2026-09-04 19:23:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/65423/steps/14/logs/stdio