1017 2011-04-29 14:10:28 +0000 Transient errors triggered by "OperationalError: database is locked" 2011-08-05 07:43:19 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 1.0 x86 Multiple RESOLVED FIXED Medium normal 1.1 1 dvhart richard.purdie josh poky.bs.watcher poky.watcher sgw --- --- --- oldest_to_newest 13518 0 dvhart 2011-04-29 14:10:28 +0000 When doing highly parallel builds I hit the following once or multiple times during a build. The more recipes it needs to parse (versus use the cache), the more often I hit it. Simply restarting the build eventually works passed it. $ time bitbake poky-image-sato-live Pseudo is not present but is required, building this first before the main build NOTE: Error expanding variable WORKDIR | ETA: 00:03:30 NOTE: Error expanding variable WORKDIR ERROR: Error parsing /home/dvhart/source/poky.git/meta/recipes-graphics/libmatchbox/libmatchbox_git.bb: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception OperationalError: database is locked ERROR: Command execution failed: Exited with 1 13578 1 dvhart 2011-05-04 16:57:32 +0000 As a data point, I tried building core-image-sato master today for atom-pc: 61018fda65fcb7a08d74b57abfe8e8809fcdaae0 I included meta and meta-yocto in meta layers. I deleted tmp and sstate-cache. BB_NUMBER_THREADS = "24" PARALLEL_MAKE = "-j 24" There was a lot of disk I/O from background rm processes. It took my 5 runs to get it to complete parsing the recipes and start running tasks. 13619 2 richard.purdie 2011-05-06 14:15:19 +0000 Crazy request but this code just changed heavily in master with changes from upstream. Could you retest please? 13622 3 dvhart 2011-05-06 14:33:43 +0000 Will do so tonight. 13627 4 dvhart 2011-05-06 17:27:16 +0000 Well, turns out I was a few commits behind on my build machine. Pulling the latest master (0788081ac7a04f94c349b2673ed6ae122aa7d1a8) and trying to build core-image-sato yields an immediate new kind of failure: Pseudo is not present but is required, building this first before the main build Traceback (most recent call last): File "/home/dvhart/source/poky.git/bitbake/bin/bitbake", line 34, in <module> import bb File "/home/dvhart/source/poky.git/bitbake/lib/bb/__init__.py", line 74, in <module> from bb import fetch2 as fetch File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 31, in <module> import bb.data, bb.persist_data, bb.utils File "/home/dvhart/source/poky.git/bitbake/lib/bb/persist_data.py", line 29, in <module> from bb.compat import total_ordering ImportError: No module named compat /home/dvhart/source/poky.git/bitbake/lib/bb/event.py:100: RuntimeWarning: Parent module 'bb' not found while handling absolute import from bb.msg import BBLogFormatter Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/home/dvhart/source/poky.git/bitbake/lib/bb/event.py", line 100, in print_ui_queue from bb.msg import BBLogFormatter File "/home/dvhart/source/poky.git/bitbake/lib/bb/__init__.py", line 74, in <module> from bb import fetch2 as fetch File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 31, in <module> import bb.data, bb.persist_data, bb.utils File "/home/dvhart/source/poky.git/bitbake/lib/bb/persist_data.py", line 29, in <module> from bb.compat import total_ordering ImportError: No module named compat Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/home/dvhart/source/poky.git/bitbake/lib/bb/event.py", line 100, in print_ui_queue from bb.msg import BBLogFormatter File "/home/dvhart/source/poky.git/bitbake/lib/bb/__init__.py", line 74, in <module> from bb import fetch2 as fetch File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 31, in <module> import bb.data, bb.persist_data, bb.utils File "/home/dvhart/source/poky.git/bitbake/lib/bb/persist_data.py", line 29, in <module> from bb.compat import total_ordering ImportError: No module named compat 13628 5 dvhart 2011-05-06 17:35:58 +0000 Bisecting the last 45 patches unfortunately yielded a pair of different sorts of failures. The one reported above as well as: Pseudo is not present but is required, building this first before the main build Traceback (most recent call last): File "/home/dvhart/source/poky.git/bitbake/bin/bitbake", line 219, in <module> ret = main() File "/home/dvhart/source/poky.git/bitbake/bin/bitbake", line 193, in main cooker = bb.cooker.BBCooker(configuration, server) File "/home/dvhart/source/poky.git/bitbake/lib/bb/cooker.py", line 82, in __init__ self.parseConfigurationFiles(self.configuration.file) File "/home/dvhart/source/poky.git/bitbake/lib/bb/cooker.py", line 628, in parseConfigurationFiles bb.fetch.fetcher_init(self.configuration.data) File "/home/dvhart/source/poky.git/bitbake/lib/bb/fetch2/__init__.py", line 225, in fetcher_init pd = persist_data.persist(d) NameError: global name 'persist_data' is not defined We should be ensuring our patch series are bisectable. 13629 6 josh 2011-05-06 18:10:29 +0000 I think this is a separate issue. We're missing a module from upstream BitBake[1] which was added by a patch from Chris Larson[2]. 1. http://git.openembedded.org/cgit.cgi/bitbake/tree/lib/bb/compat.py 2/ http://git.openembedded.org/cgit.cgi/bitbake/commit/lib/bb/compat.py?id=1190406c526c7bb7cf415867be83e0403812a7dd 13632 7 richard.purdie 2011-05-07 01:00:37 +0000 compat.py is a separate issue which has been fixed. Changes should always be bisectable, I agree but in this case I failed to merge all the patches in the correct order and didn't realise until it was too late to be able to correct the series. If you could test with the compat.py file added I'd appreciate it, thanks and sorry about this mess. 13633 8 dvhart 2011-05-07 10:09:11 +0000 Sorry about the last two comments, those should have gone to the list, obviously not related. With master as of: commit 506241c914383c20561144069ba0b0d3850ce8cc Author: Richard Purdie <richard.purdie@linuxfoundation.org> Date: Sat May 7 08:49:28 2011 +0100 bitbake: Add missing file I did 5 clean recipe parsings in a row for each of qemux86 and n450: $ rm -rf tmp; bitbake core-image-sato And saw no "database is locked" errors. Perhaps this is behind us? I suggest we close with the understand that without an explicit fix we may have just changed the race window sufficiently to mask the issue for a time. We may need to reopen. 15258 9 richard.purdie 2011-08-05 07:43:19 +0000 Closing as we're not seeing reports of these anymore. Please reopen or refile if these occur again.