<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>16515</bug_id>
          
          <creation_ts>2026-11-15 17:28:04 +0000</creation_ts>
          <short_desc>rrs_upgrade_history: a commit that fails to analyse is saved as the checkpoint and never retried</short_desc>
          <delta_ts>2026-11-15 17:36:41 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>6</classification_id>
          <classification>Yocto Project Subprojects</classification>
          <product>Recipe Reporting System (RRS)</product>
          <component>Recipe Reporting System (RRS)</component>
          <version>unspecified</version>
          <rep_platform>All</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>IN PROGRESS DESIGN</bug_status>
          <resolution></resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>---</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Tim Orling">tim.orling</reporter>
          <assigned_to name="Tim Orling">tim.orling</assigned_to>
          
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Don&apos;t know</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>107490</commentid>
    <comment_count>0</comment_count>
    <who name="Tim Orling">tim.orling</who>
    <bug_when>2026-11-15 17:28:04 +0000</bug_when>
    <thetext>rrs_upgrade_history.py saves a commit as the layer branch&apos;s checkpoint
(upgrade_rev/upgrade_date) after running upgrade_history_internal.py on it,
whether or not that run succeeded. A commit that fails is never analysed
again, and nothing shows it: the script still exits 0.

On a test instance every bitbake server that upgrade_history_internal.py
starts stayed behind as a zombie, because the container had no init
process to reap it. After about 3400 commits the container reached its
pids limit (systemd DefaultTasksMax, 3444) and every following commit
failed with:

  fatal: unable to create threaded lstat: Resource temporarily unavailable

The run carried on for two more hours and moved the checkpoints of
meta-python, meta-filesystems and meta-virtualization to the latest
commits, though about 1950 commits had not been analysed. The missing
upgrade history only came back after resetting the checkpoints by hand.

A failed commit should stop that layer branch without saving the
checkpoint, so the next run retries it, and the script should exit
non-zero. (A commit that times out is still skipped, see #16508.)

Running the RRS tools with &quot;docker run --init&quot; avoids the zombies.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>107491</commentid>
    <comment_count>1</comment_count>
    <who name="Tim Orling">tim.orling</who>
    <bug_when>2026-11-15 17:36:40 +0000</bug_when>
    <thetext>Fix: &quot;rrs_upgrade_history: don&apos;t save a failed commit as the checkpoint&quot;
(branch timo/rrs-commit-failure, on top of the #16508 timeout patch).

Tested on the test bench: in a container without --init and with
--pids-limit 80, the analysis started failing after about 70 commits with
the same &quot;unable to create threaded lstat&quot; error. meta-python stopped at
d2656956d9 and kept b71b3e83fd (the last commit that worked) as its
checkpoint; meta-filesystems and meta-virtualization also stopped at their
first failure. The script exited 1.

On the bench the RRS runs now use &quot;docker run --init&quot;, and the three
checkpoints that had moved past the failed commits were reset.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>