Bug 7562

Summary: memres bitbake: uncaught exception occured in runqueue
Product: [Build System, Metadata & Runtime] BitBake Reporter: Benjamin Esquivel <benjamin.esquivel>
Component: bitbakeAssignee: Ed Bartosh <eduard.bartosh>
Status: VERIFIED FIXED QA Contact: Cristina Agurida <cristina-danielax.agurida>
Severity: normal    
Priority: Medium+ CC: alexandru.c.georgescu, cristina-danielax.agurida, poky.bs.watcher, poky.watcher
Version: 1.9   
Target Milestone: 1.9 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
bitbake log
none
bitbake server log none

Description Benjamin Esquivel 2015-04-02 18:30:47 UTC
Created attachment 2465 [details]
bitbake log

I started a round of bb-perf testing with the memres bitbake (bb-perf serializes bitbake runs with a list of permutations of bbthread vs. make-thread) and it failed at the second attempt.

failure at the terminal (see attachments for full log):

ERROR: An uncaught exception occured in runqueue, please see the failure below:
Comment 1 Benjamin Esquivel 2015-04-02 18:31:35 UTC
Created attachment 2466 [details]
bitbake server log
Comment 2 Benjamin Esquivel 2015-04-03 02:02:01 UTC
I figured out this bug is consistent, it has failed every time and always at the second run with exactly the same error.

Trying to find a simpler reproducer.
Comment 3 Ed Bartosh 2015-04-10 11:07:22 UTC
I can't reproduce this bug.

This is what I did:
. oe-init-build-en 2000 build-memres
bitbake core-image-minimal
bitbake -c clean core-image-minimal
bitbake core-image-minimal


Please, provide more detailed instructions on how to reproduce it.
Comment 4 Ed Bartosh 2015-04-10 11:36:37 UTC
"No such file or directory" error can only happen if tmp/stamps/qemux86-poky-linux/core-image-minimal/ directory doesn't exist when calling open(stamp, "w").close()

Could it be so that bitbake -c clean core-image-minimal is running in parallel ? 
Or may be some other scripts remove tmp, tmp/stamps, tmp/stamps/qemux86-poky-linux/ or tmp/stamps/qemux86-poky-linux/core-image-minimal/ directories?
Comment 5 Benjamin Esquivel 2015-04-13 23:49:23 UTC
the script that is used for reproducing this bug is here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/contrib/bb-perf/bb-matrix.sh

it indeed deletes tmp at line 67 but it is not in parallel to bitbake running its thing.

What do you think?
Comment 6 Ed Bartosh 2015-04-14 14:23:06 UTC
Benjamin, thank you for providing clear way to reproduce the issue.

It's reproduceable with this steps:

. oe-init-build-memres
bitbake core-image-minimal
rm -rf tmp
bitbake core-image-minimal

Looks like bitbake server somehow remembers/reuses stamps from previous build.
I'll investigate further.
Comment 7 Ed Bartosh 2015-04-14 16:46:25 UTC
Here is my first attempt to fix it: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005630.html
works for me just fine. I tried it with Benjamin's script and was not able to crash bitbake.
Comment 8 Benjamin Esquivel 2015-04-14 22:51:49 UTC
Ed, is there an exception that could be trapped with the try statement?

def cached_mtime_noerror(f):
    if f not in __mtime_cache:
        try:
            __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
        except OSError: <--- HERE
            return 0
    return __mtime_cache[f]
Comment 9 Ed Bartosh 2015-04-15 09:57:56 UTC
Benjamin, the reason for the crash as I see it is that mtime cache is invalid, i.e. contains the path to the deleted file. That means that "f not in __mtime_cache" is not true (f is in the cache), so the code under that if statement is not executed.
Comment 10 Ed Bartosh 2015-04-15 09:59:06 UTC
BTW, can you can try if my fix works for you?
Comment 11 Benjamin Esquivel 2015-04-15 14:46:39 UTC
I will try your fix today and let you know my findings
Comment 12 Benjamin Esquivel 2015-04-15 20:21:16 UTC
Indeed it will fail no more with your patch.

Question, have you figured why the bitbake in server mode remembers the stamps from the previous build?
Comment 13 Ed Bartosh 2015-04-15 21:29:01 UTC
Bitbake doesn't clean mtime cache when build is done. It makes sence to do as build can be run again and cache can be reused.
The way how your script runs builds is a special case as usually tmp is not removed. So, I think invalidating mtime cache is good enough solution for this issue.
Comment 14 Benjamin Esquivel 2015-04-15 22:09:34 UTC
Thank you for the explanation, I guess I'm still missing to understand why this fails on bitbake server mode only.
Comment 15 Ed Bartosh 2015-04-16 09:32:35 UTC
it fails on resident mode because mtime cache is kept in memory and not removed as bitbake process is resident.
Comment 16 Ed Bartosh 2015-04-21 11:56:08 UTC
Second attempt to fix this bug: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005666.html

probably not the last one :)
Comment 17 Ed Bartosh 2015-04-21 15:52:51 UTC
One more attempt. Separated build mtime cache as Richard suggested: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005675.html
Comment 18 Ed Bartosh 2015-04-29 15:12:02 UTC
Should be fixed by this commit: http://cgit.openembedded.org/bitbake/commit/?id=f8590547a198a78334debdf14bf40acb50c22ecc
Comment 19 Benjamin Esquivel 2015-04-29 15:33:44 UTC
looking good! thanks for enabling me to run the bb perf benchmark.
Comment 20 Cristina Agurida 2015-06-11 11:51:24 UTC
Verified on master: 926de56da9469e3d4ea8349d8e7c9777ae6f93c1