Bug 16515

Summary: rrs_upgrade_history: a commit that fails to analyse is saved as the checkpoint and never retried
Product: [Yocto Project Subprojects] Recipe Reporting System (RRS) Reporter: Tim Orling <tim.orling>
Component: Recipe Reporting System (RRS)Assignee: Tim Orling <tim.orling>
Status: IN PROGRESS DESIGN --- QA Contact:
Severity: normal    
Priority: Medium    
Version: unspecified   
Target Milestone: ---   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Tim Orling 2026-11-15 17:28:04 UTC
rrs_upgrade_history.py saves a commit as the layer branch's checkpoint
(upgrade_rev/upgrade_date) after running upgrade_history_internal.py on it,
whether or not that run succeeded. A commit that fails is never analysed
again, and nothing shows it: the script still exits 0.

On a test instance every bitbake server that upgrade_history_internal.py
starts stayed behind as a zombie, because the container had no init
process to reap it. After about 3400 commits the container reached its
pids limit (systemd DefaultTasksMax, 3444) and every following commit
failed with:

  fatal: unable to create threaded lstat: Resource temporarily unavailable

The run carried on for two more hours and moved the checkpoints of
meta-python, meta-filesystems and meta-virtualization to the latest
commits, though about 1950 commits had not been analysed. The missing
upgrade history only came back after resetting the checkpoints by hand.

A failed commit should stop that layer branch without saving the
checkpoint, so the next run retries it, and the script should exit
non-zero. (A commit that times out is still skipped, see #16508.)

Running the RRS tools with "docker run --init" avoids the zombies.
Comment 1 Tim Orling 2026-11-15 17:36:40 UTC
Fix: "rrs_upgrade_history: don't save a failed commit as the checkpoint"
(branch timo/rrs-commit-failure, on top of the #16508 timeout patch).

Tested on the test bench: in a container without --init and with
--pids-limit 80, the analysis started failing after about 70 commits with
the same "unable to create threaded lstat" error. meta-python stopped at
d2656956d9 and kept b71b3e83fd (the last commit that worked) as its
checkpoint; meta-filesystems and meta-virtualization also stopped at their
first failure. The script exited 1.

On the bench the RRS runs now use "docker run --init", and the three
checkpoints that had moved past the failed commits were reset.