QEMU-based testing (testimage, ptest images, oe-selftest) makes a private copy of each rootfs per QEMU instance in the system temp dir (/srv/pokybuild/tmp on the autobuilder workers), named: <image>-<machine>.rootfs[-<timestamp>].<ext4|wic>.<PID> These copies are not being removed when the QEMU instance exits. They accumulate until the filesystem fills, after which later builds fail. Builds affected by /srv/pokybuild/tmp filling: https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/5057 qemuarm64-musl-ptest: https://autobuilder.yoctoproject.org/valkyrie/#/builders/109/builds/1488 https://autobuilder.yoctoproject.org/valkyrie/#/builders/109/builds/1491 https://autobuilder.yoctoproject.org/valkyrie/#/builders/109/builds/1492 Observed on workers: stream10-vk-arm1 on Oct 10th, I don't have a link. stream10-vk-arm1 (63G used): - All usage is from a single ptest run (core-image-ptest-*-qemuarm64, timestamp 20261103064638). - The disk filled partway through that run. Sorted by PID, copies are full size up to diffutils. diffstat is truncated (250M vs ~346M), and every later copy is 0 bytes. - All four subsequent runs (093920, 104045, 141342, 145852) produced only 0-byte copies. Their test results are invalid. ubuntu2404-vk-arm2 (126G used): - Same pattern, one run later. 064642 is fully populated. 093123 is full up to libconvert-asn1-perl, libconfig is truncated (290M vs 345M), and everything after it is 0 bytes. Runs 141340, 150739 and 155909 are all 0 bytes. fedora45-vk-1 x86-64 worker (126G used): - Copies date back to 2026-10-20 and the same set recurs on every oe-selftest run, so successful runs leak too, not just failed ones. Examples: - 4 x 1.8G core-image-minimal-qemux86-64 sharing one timestamp, once per run (8 runs, ~57G total) - repeating sets of core-image-minimal variants and oe-selftest-image - ~110 untimestamped .wic copies (core-image-minimal, core-image-base, core-image-minimal-mtdutils, wic-image-minimal) - 4 x 3.4G core-image-sato-sdk from testimage - There are signs this disk also filled on 2026-10-30: sato-sdk truncated to 2.3G, and a 0-byte core-image-weston copy. Copies from failed copy attempts (0-byte files) also leak, so cleanup is skipped on the error path as well. Suspected cause: the runqemu cleanup that removes the per-instance copy never runs when QEMU is driven by oeqa/qemurunner. One possibility is that qemurunner kills the runqemu process group on teardown before cleanup can run. Not yet confirmed. Workaround: the workers' tmp directories have been cleaned manually. A periodic age-based cleanup of *.rootfs*.{ext4,wic}.<PID> in /srv/pokybuild/tmp could serve as a stopgap until the leak is fixed.
I've gone ahead and added a script and a cronjob on each worker (/usr/local/sbin/clean-qemu-rootfs /etc/cron.d/clean-qemu-rootfs) to keep this problem from causing too much trouble while we track it down.
Created attachment 5254 [details] Abadoned files from oeqa qemuruns