Bug 16508 - RRS: one hung commit stalls rrs_upgrade_history.py indefinitely; add a per-commit timeout
Summary: RRS: one hung commit stalls rrs_upgrade_history.py indefinitely; add a per-co...
Status: IN PROGRESS REVIEW
Alias: None
Product: Recipe Reporting System (RRS)
Classification: Yocto Project Subprojects
Component: Recipe Reporting System (RRS) (show other bugs)
Version: unspecified
Hardware: All Multiple
: Medium normal
Target Milestone: 6.2 M2
Assignee: Tim Orling
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2026-11-15 01:04 UTC by Tim Orling
Modified: 2026-11-15 01:54 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Orling 2026-11-15 01:04:09 UTC
rrs_upgrade_history.py analyses each commit by running
upgrade_history_internal.py through utils.run_command_interruptible(),
which waits for it with no time limit. When a commit makes the parse hang,
the whole run stalls and keeps the layer index update lock until someone
kills it. Bugs 16500 and 16504 were found this way: on a test instance,
the run sat idle on one OE-Core commit for over 7 hours.

The nightly RRS job on the test instance (rrs-update.service) has no
timeout either. A timeout around the whole run (timeout(1) or systemd)
doesn't help much: it stops the run instead of skipping the bad commit,
and in the hang the stuck process was the BitBake server started by
upgrade_history_internal.py, which a plain kill of the child leaves
behind.

Proposed fix: an optional timeout in run_command_interruptible() that
kills the command's whole process group and raises
subprocess.TimeoutExpired, and a --commit-timeout option to
rrs_upgrade_history.py (default 1 hour). With it, a commit that times out
is logged as an error and skipped.
Comment 1 Tim Orling 2026-11-15 01:21:09 UTC
Patch sent to yocto-patches, under review (applies on top of the RRS series for bugs 16499-16506):
https://lore.kernel.org/all/20261115011321.58851-2-tim.orling@konsulko.com/
Comment 2 Tim Orling 2026-11-15 01:54:42 UTC
v2 sent (fixes a race in the new test; v1 is superseded):
https://lore.kernel.org/all/20261115015200.64142-2-tim.orling@konsulko.com/