rrs_upgrade_history.py saves a commit as the layer branch's checkpoint (upgrade_rev/upgrade_date) after running upgrade_history_internal.py on it, whether or not that run succeeded. A commit that fails is never analysed again, and nothing shows it: the script still exits 0. On a test instance every bitbake server that upgrade_history_internal.py starts stayed behind as a zombie, because the container had no init process to reap it. After about 3400 commits the container reached its pids limit (systemd DefaultTasksMax, 3444) and every following commit failed with: fatal: unable to create threaded lstat: Resource temporarily unavailable The run carried on for two more hours and moved the checkpoints of meta-python, meta-filesystems and meta-virtualization to the latest commits, though about 1950 commits had not been analysed. The missing upgrade history only came back after resetting the checkpoints by hand. A failed commit should stop that layer branch without saving the checkpoint, so the next run retries it, and the script should exit non-zero. (A commit that times out is still skipped, see #16508.) Running the RRS tools with "docker run --init" avoids the zombies.
Fix: "rrs_upgrade_history: don't save a failed commit as the checkpoint" (branch timo/rrs-commit-failure, on top of the #16508 timeout patch). Tested on the test bench: in a container without --init and with --pids-limit 80, the analysis started failing after about 70 commits with the same "unable to create threaded lstat" error. meta-python stopped at d2656956d9 and kept b71b3e83fd (the last commit that worked) as its checkpoint; meta-filesystems and meta-virtualization also stopped at their first failure. The script exited 1. On the bench the RRS runs now use "docker run --init", and the three checkpoints that had moved past the failed commits were reset.