Created attachment 2465 [details] bitbake log I started a round of bb-perf testing with the memres bitbake (bb-perf serializes bitbake runs with a list of permutations of bbthread vs. make-thread) and it failed at the second attempt. failure at the terminal (see attachments for full log): ERROR: An uncaught exception occured in runqueue, please see the failure below:
Created attachment 2466 [details] bitbake server log
I figured out this bug is consistent, it has failed every time and always at the second run with exactly the same error. Trying to find a simpler reproducer.
I can't reproduce this bug. This is what I did: . oe-init-build-en 2000 build-memres bitbake core-image-minimal bitbake -c clean core-image-minimal bitbake core-image-minimal Please, provide more detailed instructions on how to reproduce it.
"No such file or directory" error can only happen if tmp/stamps/qemux86-poky-linux/core-image-minimal/ directory doesn't exist when calling open(stamp, "w").close() Could it be so that bitbake -c clean core-image-minimal is running in parallel ? Or may be some other scripts remove tmp, tmp/stamps, tmp/stamps/qemux86-poky-linux/ or tmp/stamps/qemux86-poky-linux/core-image-minimal/ directories?
the script that is used for reproducing this bug is here: http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/contrib/bb-perf/bb-matrix.sh it indeed deletes tmp at line 67 but it is not in parallel to bitbake running its thing. What do you think?
Benjamin, thank you for providing clear way to reproduce the issue. It's reproduceable with this steps: . oe-init-build-memres bitbake core-image-minimal rm -rf tmp bitbake core-image-minimal Looks like bitbake server somehow remembers/reuses stamps from previous build. I'll investigate further.
Here is my first attempt to fix it: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005630.html works for me just fine. I tried it with Benjamin's script and was not able to crash bitbake.
Ed, is there an exception that could be trapped with the try statement? def cached_mtime_noerror(f): if f not in __mtime_cache: try: __mtime_cache[f] = os.stat(f)[stat.ST_MTIME] except OSError: <--- HERE return 0 return __mtime_cache[f]
Benjamin, the reason for the crash as I see it is that mtime cache is invalid, i.e. contains the path to the deleted file. That means that "f not in __mtime_cache" is not true (f is in the cache), so the code under that if statement is not executed.
BTW, can you can try if my fix works for you?
I will try your fix today and let you know my findings
Indeed it will fail no more with your patch. Question, have you figured why the bitbake in server mode remembers the stamps from the previous build?
Bitbake doesn't clean mtime cache when build is done. It makes sence to do as build can be run again and cache can be reused. The way how your script runs builds is a special case as usually tmp is not removed. So, I think invalidating mtime cache is good enough solution for this issue.
Thank you for the explanation, I guess I'm still missing to understand why this fails on bitbake server mode only.
it fails on resident mode because mtime cache is kept in memory and not removed as bitbake process is resident.
Second attempt to fix this bug: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005666.html probably not the last one :)
One more attempt. Separated build mtime cache as Richard suggested: http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005675.html
Should be fixed by this commit: http://cgit.openembedded.org/bitbake/commit/?id=f8590547a198a78334debdf14bf40acb50c22ecc
looking good! thanks for enabling me to run the bb perf benchmark.
Verified on master: 926de56da9469e3d4ea8349d8e7c9777ae6f93c1