... but when we had it lower (5s) we saw a lot of "OperationalError: database is locked" exceptions on first run. There's certainly an underlying issue here with a theory that there's an underlying issue in the python->pysqlite->sqlite stack that we trigger with our initial burst of writes to the database. The 5s timeout ensured the OperationalError exception was raised on each first run on the autobuilder infrastructure, however the issue has proven much more difficult to reproduce on a variety of local setups. It'd be great to work with upstream on this issue so that we don't have this arbitrarily large timeout value.
I think one of the reasons we're seeing this now on the autobuilder infrastructure is due to the addition of an extra build slave per autobuilder. That has slowed things down just enough to expose this issue (which we see in both bernard and master. It is most likely in edison as well.
We're starting to see this more and more on the autobuilders, e.g. http://autobuilder.pokylinux.org:8010/builders/nightly-x86/builds/286/steps/shell_67/logs/stdio. Reassigning the priority to a medium bug.
Assign to Lianhao to help on it
Created attachment 301 [details] mimic bitbake to reproduce the bug Using the attached python code (1761.py), we can mimic what happened in bitbake and reproduce this bug easily both on Ubuntu 10.04_x86_64 and openSuse 11.04_x86_64. sqlite is not designed to work with large amount of concurrent connections at the same time. However, we need to figure out a way to avoid the burst write operations and open connection operations at the same time.
Patch sent out for review http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=llu/bug1761&id=554ae4c9ee0ac70254c6ca6a5ba44db844cf86b3
Fixed in the following commit by reconnecting in the exception handler to leverage the retry count, and also drop timeout to 5 seconds. http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=1fedd166b71a4000116fcc9bd993cb23a7899adb
Verify we don't see the OperationalError issues anymore *and* have a sane timeout