<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>7562</bug_id>
          
          <creation_ts>2015-04-02 18:30:47 +0000</creation_ts>
          <short_desc>memres bitbake: uncaught exception occured in runqueue</short_desc>
          <delta_ts>2015-06-11 11:51:24 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>1.9</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>VERIFIED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>1.9 M1</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Benjamin Esquivel">benjamin.esquivel</reporter>
          <assigned_to name="Ed Bartosh">eduard.bartosh</assigned_to>
          <cc>alexandru.c.georgescu</cc>
    
    <cc>cristina-danielax.agurida</cc>
    
    <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
          
          <qa_contact name="Cristina Agurida">cristina-danielax.agurida</qa_contact>
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>50053</commentid>
    <comment_count>0</comment_count>
      <attachid>2465</attachid>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-02 18:30:47 +0000</bug_when>
    <thetext>Created attachment 2465
bitbake log

I started a round of bb-perf testing with the memres bitbake (bb-perf serializes bitbake runs with a list of permutations of bbthread vs. make-thread) and it failed at the second attempt.

failure at the terminal (see attachments for full log):

ERROR: An uncaught exception occured in runqueue, please see the failure below:</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50054</commentid>
    <comment_count>1</comment_count>
      <attachid>2466</attachid>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-02 18:31:35 +0000</bug_when>
    <thetext>Created attachment 2466
bitbake server log</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50067</commentid>
    <comment_count>2</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-03 02:02:01 +0000</bug_when>
    <thetext>I figured out this bug is consistent, it has failed every time and always at the second run with exactly the same error.

Trying to find a simpler reproducer.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50219</commentid>
    <comment_count>3</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-10 11:07:22 +0000</bug_when>
    <thetext>I can&apos;t reproduce this bug.

This is what I did:
. oe-init-build-en 2000 build-memres
bitbake core-image-minimal
bitbake -c clean core-image-minimal
bitbake core-image-minimal


Please, provide more detailed instructions on how to reproduce it.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50225</commentid>
    <comment_count>4</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-10 11:36:37 +0000</bug_when>
    <thetext>&quot;No such file or directory&quot; error can only happen if tmp/stamps/qemux86-poky-linux/core-image-minimal/ directory doesn&apos;t exist when calling open(stamp, &quot;w&quot;).close()

Could it be so that bitbake -c clean core-image-minimal is running in parallel ? 
Or may be some other scripts remove tmp, tmp/stamps, tmp/stamps/qemux86-poky-linux/ or tmp/stamps/qemux86-poky-linux/core-image-minimal/ directories?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50283</commentid>
    <comment_count>5</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-13 23:49:23 +0000</bug_when>
    <thetext>the script that is used for reproducing this bug is here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/contrib/bb-perf/bb-matrix.sh

it indeed deletes tmp at line 67 but it is not in parallel to bitbake running its thing.

What do you think?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50295</commentid>
    <comment_count>6</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-14 14:23:06 +0000</bug_when>
    <thetext>Benjamin, thank you for providing clear way to reproduce the issue.

It&apos;s reproduceable with this steps:

. oe-init-build-memres
bitbake core-image-minimal
rm -rf tmp
bitbake core-image-minimal

Looks like bitbake server somehow remembers/reuses stamps from previous build.
I&apos;ll investigate further.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50301</commentid>
    <comment_count>7</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-14 16:46:25 +0000</bug_when>
    <thetext>Here is my first attempt to fix it: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005630.html
works for me just fine. I tried it with Benjamin&apos;s script and was not able to crash bitbake.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50306</commentid>
    <comment_count>8</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-14 22:51:49 +0000</bug_when>
    <thetext>Ed, is there an exception that could be trapped with the try statement?

def cached_mtime_noerror(f):
    if f not in __mtime_cache:
        try:
            __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
        except OSError: &lt;--- HERE
            return 0
    return __mtime_cache[f]</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50310</commentid>
    <comment_count>9</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-15 09:57:56 +0000</bug_when>
    <thetext>Benjamin, the reason for the crash as I see it is that mtime cache is invalid, i.e. contains the path to the deleted file. That means that &quot;f not in __mtime_cache&quot; is not true (f is in the cache), so the code under that if statement is not executed.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50311</commentid>
    <comment_count>10</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-15 09:59:06 +0000</bug_when>
    <thetext>BTW, can you can try if my fix works for you?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50317</commentid>
    <comment_count>11</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-15 14:46:39 +0000</bug_when>
    <thetext>I will try your fix today and let you know my findings</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50324</commentid>
    <comment_count>12</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-15 20:21:16 +0000</bug_when>
    <thetext>Indeed it will fail no more with your patch.

Question, have you figured why the bitbake in server mode remembers the stamps from the previous build?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50326</commentid>
    <comment_count>13</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-15 21:29:01 +0000</bug_when>
    <thetext>Bitbake doesn&apos;t clean mtime cache when build is done. It makes sence to do as build can be run again and cache can be reused.
The way how your script runs builds is a special case as usually tmp is not removed. So, I think invalidating mtime cache is good enough solution for this issue.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50329</commentid>
    <comment_count>14</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-15 22:09:34 +0000</bug_when>
    <thetext>Thank you for the explanation, I guess I&apos;m still missing to understand why this fails on bitbake server mode only.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50332</commentid>
    <comment_count>15</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-16 09:32:35 +0000</bug_when>
    <thetext>it fails on resident mode because mtime cache is kept in memory and not removed as bitbake process is resident.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50437</commentid>
    <comment_count>16</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-21 11:56:08 +0000</bug_when>
    <thetext>Second attempt to fix this bug: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005666.html

probably not the last one :)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50449</commentid>
    <comment_count>17</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-21 15:52:51 +0000</bug_when>
    <thetext>One more attempt. Separated build mtime cache as Richard suggested: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005675.html</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50609</commentid>
    <comment_count>18</comment_count>
    <who name="Ed Bartosh">eduard.bartosh</who>
    <bug_when>2015-04-29 15:12:02 +0000</bug_when>
    <thetext>Should be fixed by this commit: http://cgit.openembedded.org/bitbake/commit/?id=f8590547a198a78334debdf14bf40acb50c22ecc</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>50613</commentid>
    <comment_count>19</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2015-04-29 15:33:44 +0000</bug_when>
    <thetext>looking good! thanks for enabling me to run the bb perf benchmark.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>51639</commentid>
    <comment_count>20</comment_count>
    <who name="Cristina Agurida">cristina-danielax.agurida</who>
    <bug_when>2015-06-11 11:51:24 +0000</bug_when>
    <thetext>Verified on master: 926de56da9469e3d4ea8349d8e7c9777ae6f93c1</thetext>
  </long_desc>
      
          <attachment
              isobsolete="0"
              ispatch="0"
              isprivate="0"
          >
            <attachid>2465</attachid>
            <date>2015-04-02 18:30:47 +0000</date>
            <delta_ts>2015-04-02 18:30:47 +0000</delta_ts>
            <desc>bitbake log</desc>
            <filename>bitbake.log</filename>
            <type>text/x-log</type>
            <size>933</size>
            <attacher name="Benjamin Esquivel">benjamin.esquivel</attacher>
            
              <data encoding="base64">V0FSTklORzogSG9zdCBkaXN0cmlidXRpb24gIm9wZW5TVVNFLXByb2plY3QtMTMuMSIgaGFzIG5v
dCBiZWVuIHZhbGlkYXRlZCB3aXRoIHRoaXMgdmVyc2lvbiBvZiB0aGUgYnVpbGQgc3lzdGVtOyB5
b3UgbWF5IHBvc3NpYmx5IGV4cGVyaWVuY2UgdW5leHBlY3RlZCBmYWlsdXJlcy4gSXQgaXMgcmVj
b21tZW5kZWQgdGhhdCB5b3UgdXNlIGEgdGVzdGVkIGRpc3RyaWJ1dGlvbi4KUGFyc2luZyByZWNp
cGVzLi4uZG9uZS4KUGFyc2luZyBvZiA4ODQgLmJiIGZpbGVzIGNvbXBsZXRlICgwIGNhY2hlZCwg
ODg0IHBhcnNlZCkuIDEyODUgdGFyZ2V0cywgNDEgc2tpcHBlZCwgMCBtYXNrZWQsIDAgZXJyb3Jz
LgpOT1RFOiBSZXNvbHZpbmcgYW55IG1pc3NpbmcgdGFzayBxdWV1ZSBkZXBlbmRlbmNpZXMKCkJ1
aWxkIENvbmZpZ3VyYXRpb246CkJCX1ZFUlNJT04gICAgICAgID0gIjEuMjcuMCIKQlVJTERfU1lT
ICAgICAgICAgPSAieDg2XzY0LWxpbnV4IgpOQVRJVkVMU0JTVFJJTkcgICA9ICJvcGVuU1VTRS1w
cm9qZWN0LTEzLjEiClRBUkdFVF9TWVMgICAgICAgID0gImk1ODYtcG9reS1saW51eCIKTUFDSElO
RSAgICAgICAgICAgPSAicWVtdXg4NiIKRElTVFJPICAgICAgICAgICAgPSAicG9reSIKRElTVFJP
X1ZFUlNJT04gICAgPSAiMS44K3NuYXBzaG90LTIwMTUwNDAxIgpUVU5FX0ZFQVRVUkVTICAgICA9
ICJtMzIgaTU4NiIKVEFSR0VUX0ZQVSAgICAgICAgPSAiIgptZXRhICAgICAgICAgICAgICAKbWV0
YS15b2N0byAgICAgICAgCm1ldGEteW9jdG8tYnNwICAgID0gIm1hc3Rlcjo3MjNjNjRiODNkNGU4
NDVhMDc2NjdkNDZhZDk4NGI5OWE0YTMwODQ3IgoKTk9URTogUHJlcGFyaW5nIFJ1blF1ZXVlCkVS
Uk9SOiBBbiB1bmNhdWdodCBleGNlcHRpb24gb2NjdXJlZCBpbiBydW5xdWV1ZSwgcGxlYXNlIHNl
ZSB0aGUgZmFpbHVyZSBiZWxvdzoK
</data>

          </attachment>
          <attachment
              isobsolete="0"
              ispatch="0"
              isprivate="0"
          >
            <attachid>2466</attachid>
            <date>2015-04-02 18:31:35 +0000</date>
            <delta_ts>2015-04-02 18:31:35 +0000</delta_ts>
            <desc>bitbake server log</desc>
            <filename>bitbake-cookerdaemon.log</filename>
            <type>text/x-log</type>
            <size>1068</size>
            <attacher name="Benjamin Esquivel">benjamin.esquivel</attacher>
            
              <data encoding="base64">VHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgogIEZpbGUgIi9ob21lL2Jlc3F1aXZl
L3dvcmtkaXIvcG9reS9iaXRiYWtlL2xpYi9iYi9zZXJ2ZXIveG1scnBjLnB5IiwgbGluZSAyNDAs
IGluIF9zZXJ2ZV9mb3JldmVyCiAgICByZXR2YWwgPSBmdW5jdGlvbihzZWxmLCBkYXRhLCBGYWxz
ZSkKICBGaWxlICIvaG9tZS9iZXNxdWl2ZS93b3JrZGlyL3Bva3kvYml0YmFrZS9saWIvYmIvY29v
a2VyLnB5IiwgbGluZSAxMzMwLCBpbiBidWlsZFRhcmdldHNJZGxlCiAgICByZXR2YWwgPSBycS5l
eGVjdXRlX3J1bnF1ZXVlKCkKICBGaWxlICIvaG9tZS9iZXNxdWl2ZS93b3JrZGlyL3Bva3kvYml0
YmFrZS9saWIvYmIvcnVucXVldWUucHkiLCBsaW5lIDEwOTQsIGluIGV4ZWN1dGVfcnVucXVldWUK
ICAgIHJldHVybiBzZWxmLl9leGVjdXRlX3J1bnF1ZXVlKCkKICBGaWxlICIvaG9tZS9iZXNxdWl2
ZS93b3JrZGlyL3Bva3kvYml0YmFrZS9saWIvYmIvcnVucXVldWUucHkiLCBsaW5lIDEwNTAsIGlu
IF9leGVjdXRlX3J1bnF1ZXVlCiAgICBzZWxmLnJxZXhlID0gUnVuUXVldWVFeGVjdXRlU2NlbmVx
dWV1ZShzZWxmKQogIEZpbGUgIi9ob21lL2Jlc3F1aXZlL3dvcmtkaXIvcG9reS9iaXRiYWtlL2xp
Yi9iYi9ydW5xdWV1ZS5weSIsIGxpbmUgMTgwMiwgaW4gX19pbml0X18KICAgIGJiLmJ1aWxkLm1h
a2Vfc3RhbXAodGFza25hbWUgKyAiX3NldHNjZW5lIiwgc2VsZi5ycWRhdGEuZGF0YUNhY2hlLCBm
bikKICBGaWxlICIvaG9tZS9iZXNxdWl2ZS93b3JrZGlyL3Bva3kvYml0YmFrZS9saWIvYmIvYnVp
bGQucHkiLCBsaW5lIDU5MSwgaW4gbWFrZV9zdGFtcAogICAgb3BlbihzdGFtcCwgInciKS5jbG9z
ZSgpCklPRXJyb3I6IFtFcnJubyAyXSBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5OiAnL2hvbWUv
YmVzcXVpdmUvd29ya2Rpci9wb2t5L2J1aWxkL3RtcC9zdGFtcHMvcWVtdXg4Ni1wb2t5LWxpbnV4
L2NvcmUtaW1hZ2UtbWluaW1hbC8xLjAtcjAuZG9fcG9wdWxhdGVfc3lzcm9vdF9zZXRzY2VuZS5l
N2Q2MjVjN2FlYTQ1YWNhMTQ2MDI2NDlhMDUyY2UyNC5xZW11eDg2JwoK
</data>

          </attachment>
      

    </bug>

</bugzilla>