Bug 16515 - rrs_upgrade_history: a commit that fails to analyse is saved as the checkpoint and never retried
Summary: rrs_upgrade_history: a commit that fails to analyse is saved as the checkpoin...
Status: IN PROGRESS DESIGN
Alias: None
Product: Recipe Reporting System (RRS)
Classification: Yocto Project Subprojects
Component: Recipe Reporting System (RRS) (show other bugs)
Version: unspecified
Hardware: All Multiple
: Medium normal
Target Milestone: ---
Assignee: Tim Orling
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2026-11-15 17:28 UTC by Tim Orling
Modified: 2026-11-15 17:36 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Orling 2026-11-15 17:28:04 UTC
rrs_upgrade_history.py saves a commit as the layer branch's checkpoint
(upgrade_rev/upgrade_date) after running upgrade_history_internal.py on it,
whether or not that run succeeded. A commit that fails is never analysed
again, and nothing shows it: the script still exits 0.

On a test instance every bitbake server that upgrade_history_internal.py
starts stayed behind as a zombie, because the container had no init
process to reap it. After about 3400 commits the container reached its
pids limit (systemd DefaultTasksMax, 3444) and every following commit
failed with:

  fatal: unable to create threaded lstat: Resource temporarily unavailable

The run carried on for two more hours and moved the checkpoints of
meta-python, meta-filesystems and meta-virtualization to the latest
commits, though about 1950 commits had not been analysed. The missing
upgrade history only came back after resetting the checkpoints by hand.

A failed commit should stop that layer branch without saving the
checkpoint, so the next run retries it, and the script should exit
non-zero. (A commit that times out is still skipped, see #16508.)

Running the RRS tools with "docker run --init" avoids the zombies.
Comment 1 Tim Orling 2026-11-15 17:36:40 UTC
Fix: "rrs_upgrade_history: don't save a failed commit as the checkpoint"
(branch timo/rrs-commit-failure, on top of the #16508 timeout patch).

Tested on the test bench: in a container without --init and with
--pids-limit 80, the analysis started failing after about 70 commits with
the same "unable to create threaded lstat" error. meta-python stopped at
d2656956d9 and kept b71b3e83fd (the last commit that worked) as its
checkpoint; meta-filesystems and meta-virtualization also stopped at their
first failure. The script exited 1.

On the bench the RRS runs now use "docker run --init", and the three
checkpoints that had moved past the failed commits were reset.