Bug 7562 - memres bitbake: uncaught exception occured in runqueue
Summary: memres bitbake: uncaught exception occured in runqueue
Status: VERIFIED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 1.9
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 1.9 M1
Assignee: Ed Bartosh
QA Contact: Cristina Agurida
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-04-02 18:30 UTC by Benjamin Esquivel
Modified: 2015-06-11 11:51 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
bitbake log (933 bytes, text/x-log)
2015-04-02 18:30 UTC, Benjamin Esquivel
no flags Details
bitbake server log (1.04 KB, text/x-log)
2015-04-02 18:31 UTC, Benjamin Esquivel
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Benjamin Esquivel 2015-04-02 18:30:47 UTC
Created attachment 2465 [details]
bitbake log

I started a round of bb-perf testing with the memres bitbake (bb-perf serializes bitbake runs with a list of permutations of bbthread vs. make-thread) and it failed at the second attempt.

failure at the terminal (see attachments for full log):

ERROR: An uncaught exception occured in runqueue, please see the failure below:
Comment 1 Benjamin Esquivel 2015-04-02 18:31:35 UTC
Created attachment 2466 [details]
bitbake server log
Comment 2 Benjamin Esquivel 2015-04-03 02:02:01 UTC
I figured out this bug is consistent, it has failed every time and always at the second run with exactly the same error.

Trying to find a simpler reproducer.
Comment 3 Ed Bartosh 2015-04-10 11:07:22 UTC
I can't reproduce this bug.

This is what I did:
. oe-init-build-en 2000 build-memres
bitbake core-image-minimal
bitbake -c clean core-image-minimal
bitbake core-image-minimal


Please, provide more detailed instructions on how to reproduce it.
Comment 4 Ed Bartosh 2015-04-10 11:36:37 UTC
"No such file or directory" error can only happen if tmp/stamps/qemux86-poky-linux/core-image-minimal/ directory doesn't exist when calling open(stamp, "w").close()

Could it be so that bitbake -c clean core-image-minimal is running in parallel ? 
Or may be some other scripts remove tmp, tmp/stamps, tmp/stamps/qemux86-poky-linux/ or tmp/stamps/qemux86-poky-linux/core-image-minimal/ directories?
Comment 5 Benjamin Esquivel 2015-04-13 23:49:23 UTC
the script that is used for reproducing this bug is here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/contrib/bb-perf/bb-matrix.sh

it indeed deletes tmp at line 67 but it is not in parallel to bitbake running its thing.

What do you think?
Comment 6 Ed Bartosh 2015-04-14 14:23:06 UTC
Benjamin, thank you for providing clear way to reproduce the issue.

It's reproduceable with this steps:

. oe-init-build-memres
bitbake core-image-minimal
rm -rf tmp
bitbake core-image-minimal

Looks like bitbake server somehow remembers/reuses stamps from previous build.
I'll investigate further.
Comment 7 Ed Bartosh 2015-04-14 16:46:25 UTC
Here is my first attempt to fix it: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005630.html
works for me just fine. I tried it with Benjamin's script and was not able to crash bitbake.
Comment 8 Benjamin Esquivel 2015-04-14 22:51:49 UTC
Ed, is there an exception that could be trapped with the try statement?

def cached_mtime_noerror(f):
    if f not in __mtime_cache:
        try:
            __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
        except OSError: <--- HERE
            return 0
    return __mtime_cache[f]
Comment 9 Ed Bartosh 2015-04-15 09:57:56 UTC
Benjamin, the reason for the crash as I see it is that mtime cache is invalid, i.e. contains the path to the deleted file. That means that "f not in __mtime_cache" is not true (f is in the cache), so the code under that if statement is not executed.
Comment 10 Ed Bartosh 2015-04-15 09:59:06 UTC
BTW, can you can try if my fix works for you?
Comment 11 Benjamin Esquivel 2015-04-15 14:46:39 UTC
I will try your fix today and let you know my findings
Comment 12 Benjamin Esquivel 2015-04-15 20:21:16 UTC
Indeed it will fail no more with your patch.

Question, have you figured why the bitbake in server mode remembers the stamps from the previous build?
Comment 13 Ed Bartosh 2015-04-15 21:29:01 UTC
Bitbake doesn't clean mtime cache when build is done. It makes sence to do as build can be run again and cache can be reused.
The way how your script runs builds is a special case as usually tmp is not removed. So, I think invalidating mtime cache is good enough solution for this issue.
Comment 14 Benjamin Esquivel 2015-04-15 22:09:34 UTC
Thank you for the explanation, I guess I'm still missing to understand why this fails on bitbake server mode only.
Comment 15 Ed Bartosh 2015-04-16 09:32:35 UTC
it fails on resident mode because mtime cache is kept in memory and not removed as bitbake process is resident.
Comment 16 Ed Bartosh 2015-04-21 11:56:08 UTC
Second attempt to fix this bug: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005666.html

probably not the last one :)
Comment 17 Ed Bartosh 2015-04-21 15:52:51 UTC
One more attempt. Separated build mtime cache as Richard suggested: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005675.html
Comment 18 Ed Bartosh 2015-04-29 15:12:02 UTC
Should be fixed by this commit: http://cgit.openembedded.org/bitbake/commit/?id=f8590547a198a78334debdf14bf40acb50c22ecc
Comment 19 Benjamin Esquivel 2015-04-29 15:33:44 UTC
looking good! thanks for enabling me to run the bb perf benchmark.
Comment 20 Cristina Agurida 2015-06-11 11:51:24 UTC
Verified on master: 926de56da9469e3d4ea8349d8e7c9777ae6f93c1