Bug 16487 - runqemu/oeqa leaks per-instance rootfs copies (<image>.<ext4|wic>.<PID>) in /srv/pokybuild/tmp, filling the tmpfs mount and breaking ptest builds
Summary: runqemu/oeqa leaks per-instance rootfs copies (<image>.<ext4|wic>.<PID>) in /...
Status: NEW
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Undecided normal
Target Milestone: 0.0.0
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2026-11-03 16:38 UTC by Michael Halstead
Modified: 2026-11-03 18:08 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Abadoned files from oeqa qemuruns (169.39 KB, text/plain)
2026-11-03 16:58 UTC, Michael Halstead
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Halstead 2026-11-03 16:38:26 UTC
QEMU-based testing (testimage, ptest images, oe-selftest) makes a private
copy of each rootfs per QEMU instance in the system temp dir
(/srv/pokybuild/tmp on the autobuilder workers), named:

  <image>-<machine>.rootfs[-<timestamp>].<ext4|wic>.<PID>

These copies are not being removed when the QEMU instance exits. They
accumulate until the filesystem fills, after which later builds fail.

Builds affected by /srv/pokybuild/tmp filling:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/5057

qemuarm64-musl-ptest:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/109/builds/1488
https://autobuilder.yoctoproject.org/valkyrie/#/builders/109/builds/1491
https://autobuilder.yoctoproject.org/valkyrie/#/builders/109/builds/1492

Observed on workers:

stream10-vk-arm1 on Oct 10th, I don't have a link.

stream10-vk-arm1 (63G used):
- All usage is from a single ptest run (core-image-ptest-*-qemuarm64,
  timestamp 20261103064638).
- The disk filled partway through that run. Sorted by PID, copies are
  full size up to diffutils. diffstat is truncated (250M vs ~346M), and
  every later copy is 0 bytes.
- All four subsequent runs (093920, 104045, 141342, 145852) produced
  only 0-byte copies. Their test results are invalid.

ubuntu2404-vk-arm2 (126G used):
- Same pattern, one run later. 064642 is fully populated. 093123 is
  full up to libconvert-asn1-perl, libconfig is truncated (290M vs
  345M), and everything after it is 0 bytes. Runs 141340, 150739 and
  155909 are all 0 bytes.

fedora45-vk-1 x86-64 worker (126G used):
- Copies date back to 2026-10-20 and the same set recurs on every
  oe-selftest run, so successful runs leak too, not just failed ones.
  Examples:
  - 4 x 1.8G core-image-minimal-qemux86-64 sharing one timestamp, once
    per run (8 runs, ~57G total)
  - repeating sets of core-image-minimal variants and
    oe-selftest-image
  - ~110 untimestamped .wic copies (core-image-minimal,
    core-image-base, core-image-minimal-mtdutils, wic-image-minimal)
  - 4 x 3.4G core-image-sato-sdk from testimage
- There are signs this disk also filled on 2026-10-30: sato-sdk
  truncated to 2.3G, and a 0-byte core-image-weston copy.

Copies from failed copy attempts (0-byte files) also leak, so cleanup
is skipped on the error path as well.

Suspected cause: the runqemu cleanup that removes the per-instance copy
never runs when QEMU is driven by oeqa/qemurunner. One possibility is
that qemurunner kills the runqemu process group on teardown before
cleanup can run. Not yet confirmed.

Workaround: the workers' tmp directories have been cleaned manually.
A periodic age-based cleanup of *.rootfs*.{ext4,wic}.<PID> in
/srv/pokybuild/tmp could serve as a stopgap until the leak is fixed.
Comment 1 Michael Halstead 2026-11-03 16:52:04 UTC
I've gone ahead and added a script and a cronjob on each worker (/usr/local/sbin/clean-qemu-rootfs /etc/cron.d/clean-qemu-rootfs) to keep this problem from causing too much trouble while we track it down.
Comment 2 Michael Halstead 2026-11-03 16:58:12 UTC
Created attachment 5254 [details]
Abadoned files from oeqa qemuruns