<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>14023</bug_id>
          
          <creation_ts>2020-08-25 15:17:44 +0000</creation_ts>
          <short_desc>oe-selftest doesn&apos;t work with BB_SERVER_TIMEOUT=60</short_desc>
          <delta_ts>2022-04-17 21:45:59 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>OE-Core</product>
          <component>core</component>
          <version>3.2</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>4.0 M4</target_milestone>
          
          <blocked>13023</blocked>
          <everconfirmed>1</everconfirmed>
          <reporter name="Richard Purdie">richard.purdie</reporter>
          <assigned_to name="Pavel Zhukov">pavel</assigned_to>
          <cc>meta.mr.watcher</cc>
    
    <cc>meta.watcher</cc>
    
    <cc>pavel</cc>
    
    <cc>randy.macleod</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>88035</commentid>
    <comment_count>0</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2020-08-25 15:17:44 +0000</bug_when>
    <thetext>This is probably the last remaining issue for enabling memory resident bitbake by default. Example failed selftest runs with it set:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1289
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1277
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1273
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1294</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>88048</commentid>
    <comment_count>1</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2020-08-27 08:09:07 +0000</bug_when>
    <thetext>Depends on memory resident bitbake</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92672</commentid>
    <comment_count>2</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-02-25 11:58:05 +0000</bug_when>
    <thetext>https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3188</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92673</commentid>
    <comment_count>3</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-02-25 12:48:31 +0000</bug_when>
    <thetext>https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3215
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3178
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3221</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92725</commentid>
    <comment_count>4</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-03 22:54:54 +0000</bug_when>
    <thetext>Re-run and updated failures from master:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3240
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/3199
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3211
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3242</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92835</commentid>
    <comment_count>5</comment_count>
    <who name="Pavel Zhukov">pavel</who>
    <bug_when>2022-03-23 07:59:01 +0000</bug_when>
    <thetext>Progress report:

Steps to reproduce: 
1) To save time: Delete all tests but devtool.DevtoolUpgradeTests.test_devtool_upgrade and devtool.DevtoolUpgradeTests.test_devtool_upgrade_git from the devtool.DevtoolUpgradeTests testsuite. 

2) Set BB_SERVER_TIMEOUT to 60 (any value from 10 to 60 actually).

3)  rm -rf ../build-st/ &amp;&amp; BB_SERVER_TIMEOUT=10 oe-selftest -K   --run-tests devtool.DevtoolUpgradeTests 


Some findings:
Only second test fails. For example:

2022-03-22 19:01:00,563 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade: PASSED (236.70s)
2022-03-22 19:01:00,563 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade_git: FAILED (8.75s)


2022-03-22 19:39:02,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_upgrade_git: PASSED (227.88s)
2022-03-22 19:39:02,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolUpgradeTests.test_devtool_xupgrade: FAILED (8.95s)  &lt;== NOTE renamed to xupgrade to change the order of execution

I&apos;ve tracked it down to this line: https://github.com/yoctoproject/poky/blob/master/scripts/lib/devtool/standard.py#L603

build_targets is called with &quot;devtool-upgrade-test[12]&quot; &quot;do_patch&quot; parameters but actual do_fetch -&gt; do_unpack -&gt; do_patch is not called (properly cached somewhere) down in bitbake server.

WARNING: [&apos;buildTargets&apos;, [&apos;devtool-upgrade-test1&apos;], &apos;do_patch&apos;] # &lt;=             commandline debug output tinfoil.py result = self.server_connection.connection.runCommand(commandline)

NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = &quot;1.53.0&quot;
BUILD_SYS            = &quot;x86_64-linux&quot;
NATIVELSBSTRING      = &quot;universal&quot;
TARGET_SYS           = &quot;x86_64-poky-linux&quot;
MACHINE              = &quot;qemux86-64&quot;
DISTRO               = &quot;poky&quot;
DISTRO_VERSION       = &quot;3.4+snapshot-ae5f87700713e184fcca7d6bcdc5b86267cdce3b&quot;
TUNE_FEATURES        = &quot;m64 core2&quot;
TARGET_FPU           = &quot;&quot;
meta                 
meta-poky            
meta-yocto-bsp       = &quot;master:ae5f87700713e184fcca7d6bcdc5b86267cdce3b&quot;
meta-selftest        = &quot;master:db2fb814b7f3cd5ae095d05246455df5a1b004e3&quot;
workspace            = &quot;&lt;unknown&gt;:&lt;unknown&gt;&quot;

Initialising tasks...done.
Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 20 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 93 tasks of which 90 didn&apos;t need to be rerun and all succeeded.
ERROR: Something went wrong with source extraction - the devtool-source class was not active or did not function correctly:
[Errno 2] No such file or directory: &apos;/mnt/builds/yocto/build-st/tmp/work/core2-64-poky-linux/devtool-upgrade-test1/1.5.3-r5/devtooltmp-ke8zieoj/initial_rev&apos;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92874</commentid>
    <comment_count>6</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-25 15:16:53 +0000</bug_when>
    <thetext>Thanks for this, narrowing it down to a test case like this is incredibly helpful.

I did some work on trying to find the problem and initially I also thought it was some mtime cache issue. It turns out it was an inotify issue (which we use to monitor whether files have changed).

The issue is the removal of the workspace, then the addition of it back. Since we lost the inotify monitoring when the file was deleted and didn&apos;t add it back, chages would go undetected causing the bug as things didn&apos;t reparse.

I&apos;ve sent out a patch which fixes this. I&apos;m very happy to have this fixed as it should help a class of bugs where the server is slow to shutdown and is reused instead of restarted.

Whether it is the cause of all the issues in selftest I&apos;m not sure but we can test and see.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92875</commentid>
    <comment_count>7</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-25 22:44:49 +0000</bug_when>
    <thetext>After this is applied there are still failures but fewer of them. We&apos;ll probably need to identify further smaller test cases to analyse and keep going:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3350/steps/14/logs/stdio</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92876</commentid>
    <comment_count>8</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-25 23:02:24 +0000</bug_when>
    <thetext>FWIW the next reproduder I found which fails is oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_showrecipes devtool.DevtoolModifyTests.test_devtool_buildclean  -j 1 -K</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92886</commentid>
    <comment_count>9</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-30 21:08:32 +0000</bug_when>
    <thetext>I figured out some further issues with my previous inotify fix. That brings us to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3377/steps/14/logs/stdio

which at least doesn&apos;t have devtool failures...</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92905</commentid>
    <comment_count>10</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-31 16:59:37 +0000</bug_when>
    <thetext>With proposed fixes to sys.modules and sys.path, this was reduced to:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3382/steps/15/logs/stdio</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92909</commentid>
    <comment_count>11</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-03-31 20:49:13 +0000</bug_when>
    <thetext>First successful build: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3386 :)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>92926</commentid>
    <comment_count>12</comment_count>
    <who name="Pavel Zhukov">pavel</who>
    <bug_when>2022-04-06 06:48:52 +0000</bug_when>
    <thetext>from the IRC RP:  there is at least one intermittent issue which could be from memres: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3400/steps/14/logs/stdio</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>93113</commentid>
    <comment_count>13</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2022-04-17 21:45:59 +0000</bug_when>
    <thetext>The PRServ issues is now in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14786 and we&apos;ve not seen any other issues since the inotify and sys.modules and sys.path fixes were merged. I think we can close this one, thanks!</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>