16262 2026-04-23 07:46:54 +0000 AB-INT: indexing is sometimes too long (Couldn't obtain the lock) 2026-09-06 16:13:16 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple RESOLVED WORKSFORME AB-INT Medium normal 6.1 1 mathieu.dubois-briand richard.purdie infras.ab.watcher Infras.watcher mhalstead paul randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 105198 0 mathieu.dubois-briand 2026-04-23 07:46:54 +0000 We recently saw some indexing jobs failing with: Couldn't obtain the lock /srv/autobuilder/valkyrie.yocto.io/pub/non-release/indexing.lock: [Errno 11] Resource temporarily unavailable This is probably because the job before was still running. Looking at all indexing jobs on the autobuilder, we can see some builds are way longer than others. Typical time is about 2 minutes, but some jobs take more than 20 minutes. As an example, builds 52730 and 52731 failed, because of build 52729. 52729 38:33 52730 24:14 52731 9:03 105202 1 mathieu.dubois-briand 2026-04-23 07:52:45 +0000 indexing opensuse160-vk-1 master completed at 2026-04-22 11:09:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52730/steps/12/logs/stdio indexing fedora43-vk-2 master completed at 2026-04-22 11:09:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52731/steps/12/logs/stdio indexing debian13-vk-2 master completed at 2026-04-22 17:02:18+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52754/steps/12/logs/stdio 105220 2 randy.macleod 2026-04-23 14:51:16 +0000 Slow runs may be due to NFS but that's a guess. Can we fix the slow runs? 105223 3 ross.burton 2026-04-23 14:58:48 +0000 Looking at https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/52729: Generate non-release index: 26m Extract ptest logs: 7m Populate dashboard: 16s The lockfile is obtained and released in the "Generate non-release index" task, which is actually y-a-h/scripts/generate-testresult-index.py. That script does just iterate a lot of files so possibly this is "just" the NFS server struggling. 105246 4 randy.macleod 2026-04-23 17:53:14 +0000 We do generate lots of logs such as: https://valkyrie.yocto.io/pub/non-release/20260422-99/testresults/qemux86/intermittent_failure_host_data/ and I'm sure more from ptests, etc. Michael, if you collect the FS usage (du -options | zstd > du-log.zst ? ) during maintenance for whatever dir is behind: https://valkyrie.yocto.io/pub/non-release/ I don't want to do it while there are builds running. 105288 5 mhalstead 2026-04-29 00:17:57 +0000 We are starting with 719G total in pub/non-release. I can start collecting and logging regularly and add additional detail as well. 105447 6 paul 2026-05-14 15:28:23 +0000 We have done some cleanup on the NAS so that there are less files in the directories indexing is traversing, we haven't seen this recur so we can go ahead and close for now. 106151 7 mathieu.dubois-briand 2026-07-10 13:39:31 +0000 indexing debian12-vk-4 master completed at 2026-07-09 21:17:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/60107/steps/12/logs/stdio 106495 8 mathieu.dubois-briand 2026-08-11 14:54:48 +0000 indexing debian11-vk-1 master completed at 2026-08-10 18:46:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/63106/steps/15/logs/stdio 106743 9 mathieu.dubois-briand 2026-09-06 16:13:16 +0000 indexing ubuntu2510-vk-1 master completed at 2026-09-04 19:23:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/65423/steps/14/logs/stdio