Bug 16508

Summary: RRS: one hung commit stalls rrs_upgrade_history.py indefinitely; add a per-commit timeout
Product: [Yocto Project Subprojects] Recipe Reporting System (RRS) Reporter: Tim Orling <tim.orling>
Component: Recipe Reporting System (RRS)Assignee: Tim Orling <tim.orling>
Status: IN PROGRESS REVIEW --- QA Contact:
Severity: normal    
Priority: Medium    
Version: unspecified   
Target Milestone: 6.2 M2   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Tim Orling 2026-11-15 01:04:09 UTC
rrs_upgrade_history.py analyses each commit by running
upgrade_history_internal.py through utils.run_command_interruptible(),
which waits for it with no time limit. When a commit makes the parse hang,
the whole run stalls and keeps the layer index update lock until someone
kills it. Bugs 16500 and 16504 were found this way: on a test instance,
the run sat idle on one OE-Core commit for over 7 hours.

The nightly RRS job on the test instance (rrs-update.service) has no
timeout either. A timeout around the whole run (timeout(1) or systemd)
doesn't help much: it stops the run instead of skipping the bad commit,
and in the hang the stuck process was the BitBake server started by
upgrade_history_internal.py, which a plain kill of the child leaves
behind.

Proposed fix: an optional timeout in run_command_interruptible() that
kills the command's whole process group and raises
subprocess.TimeoutExpired, and a --commit-timeout option to
rrs_upgrade_history.py (default 1 hour). With it, a commit that times out
is logged as an error and skipped.
Comment 1 Tim Orling 2026-11-15 01:21:09 UTC
Patch sent to yocto-patches, under review (applies on top of the RRS series for bugs 16499-16506):
https://lore.kernel.org/all/20261115011321.58851-2-tim.orling@konsulko.com/
Comment 2 Tim Orling 2026-11-15 01:54:42 UTC
v2 sent (fixes a race in the new test; v1 is superseded):
https://lore.kernel.org/all/20261115015200.64142-2-tim.orling@konsulko.com/