1761 2011-11-08 12:54:38 +0000 30s timeout on SQLite connections seems arbitrarily large 2012-04-17 23:33:46 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 1.2 x86 Multiple VERIFIED FIXED Medium normal 1.2 1 josh lianhao.lu elizabeth.flanagan poky.bs.watcher poky.watcher richard.purdie sgw shane.wang --- --- --- oldest_to_newest 17157 0 josh 2011-11-08 12:54:38 +0000 ... but when we had it lower (5s) we saw a lot of "OperationalError: database is locked" exceptions on first run. There's certainly an underlying issue here with a theory that there's an underlying issue in the python->pysqlite->sqlite stack that we trigger with our initial burst of writes to the database. The 5s timeout ensured the OperationalError exception was raised on each first run on the autobuilder infrastructure, however the issue has proven much more difficult to reproduce on a variety of local setups. It'd be great to work with upstream on this issue so that we don't have this arbitrarily large timeout value. 17212 1 elizabeth.flanagan 2011-11-15 10:52:00 +0000 I think one of the reasons we're seeing this now on the autobuilder infrastructure is due to the addition of an extra build slave per autobuilder. That has slowed things down just enough to expose this issue (which we see in both bernard and master. It is most likely in edison as well. 17712 2 richard.purdie 2011-12-18 03:39:10 +0000 We're starting to see this more and more on the autobuilders, e.g. http://autobuilder.pokylinux.org:8010/builders/nightly-x86/builds/286/steps/shell_67/logs/stdio. Reassigning the priority to a medium bug. 17714 3 shane.wang 2011-12-18 22:43:19 +0000 Assign to Lianhao to help on it 17719 4 301 lianhao.lu 2011-12-19 02:11:23 +0000 Created attachment 301 mimic bitbake to reproduce the bug Using the attached python code (1761.py), we can mimic what happened in bitbake and reproduce this bug easily both on Ubuntu 10.04_x86_64 and openSuse 11.04_x86_64. sqlite is not designed to work with large amount of concurrent connections at the same time. However, we need to figure out a way to avoid the burst write operations and open connection operations at the same time. 18155 5 lianhao.lu 2012-01-15 20:11:22 +0000 Patch sent out for review http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=llu/bug1761&id=554ae4c9ee0ac70254c6ca6a5ba44db844cf86b3 18233 6 lianhao.lu 2012-01-18 17:11:22 +0000 Fixed in the following commit by reconnecting in the exception handler to leverage the retry count, and also drop timeout to 5 seconds. http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=1fedd166b71a4000116fcc9bd993cb23a7899adb 20581 7 josh 2012-04-17 23:33:46 +0000 Verify we don't see the OperationalError issues anymore *and* have a sane timeout 301 2011-12-19 02:11:23 +0000 2011-12-19 02:11:23 +0000 mimic bitbake to reproduce the bug 1761.py text/x-python 4304 lianhao.lu aW1wb3J0IHNxbGl0ZTMKaW1wb3J0IHN5cywgcGRiLCBvcwppbXBvcnQgY29sbGVjdGlvbnMKaW1w b3J0IHJhbmRvbQppbXBvcnQgdGltZQppbXBvcnQgdGhyZWFkaW5nCgpjbGFzcyBTUUxUYWJsZShj b2xsZWN0aW9ucy5NdXRhYmxlTWFwcGluZyk6CiAgICAiIiJPYmplY3QgcmVwcmVzZW50aW5nIGEg dGFibGUvZG9tYWluIGluIHRoZSBkYXRhYmFzZSIiIgogICAgZGVmIF9faW5pdF9fKHNlbGYsIGN1 cnNvciwgdGFibGUpOgogICAgICAgIHNlbGYuY3Vyc29yID0gY3Vyc29yCiAgICAgICAgc2VsZi50 YWJsZSA9IHRhYmxlCgogICAgICAgIHNlbGYuX2V4ZWN1dGUoIkNSRUFURSBUQUJMRSBJRiBOT1Qg RVhJU1RTICVzKGtleSBURVhULCB2YWx1ZSBURVhUKTsiCiAgICAgICAgICAgICAgICAgICAgICAl IHRhYmxlKQoKICAgIGRlZiBfZXhlY3V0ZShzZWxmLCAqcXVlcnkpOgogICAgICAgICIiIkV4ZWN1 dGUgYSBxdWVyeSwgd2FpdGluZyB0byBhY3F1aXJlIGEgbG9jayBpZiBuZWNlc3NhcnkiIiIKICAg ICAgICBjb3VudCA9IDAKICAgICAgICB0aW1lMSA9IHRpbWUudGltZSgpCiAgICAgICAgd2hpbGUg VHJ1ZToKICAgICAgICAgICAgdHJ5OgogICAgICAgICAgICAgICAgcmV0dXJuIHNlbGYuY3Vyc29y LmV4ZWN1dGUoKnF1ZXJ5KQogICAgICAgICAgICBleGNlcHQgc3FsaXRlMy5PcGVyYXRpb25hbEVy cm9yIGFzIGV4YzoKICAgICAgICAgICAgICAgIGlmICdkYXRhYmFzZSBpcyBsb2NrZWQnIGluIHN0 cihleGMpIGFuZCBjb3VudCA8IDUwMDoKICAgICAgICAgICAgICAgICAgICAjcHJpbnQgInRyeSAl ZCB0aW1lcyBhdCAlZlxuIiAlIChjb3VudCwgdGltZS50aW1lKCkpCiAgICAgICAgICAgICAgICAg ICAgY291bnQgPSBjb3VudCArIDEKICAgICAgICAgICAgICAgICAgICBjb250aW51ZQogICAgICAg ICAgICAgICAgdGltZTIgPSB0aW1lLnRpbWUoKQogICAgICAgICAgICAgICAgcHJpbnQoInJhaXNl IGV4Y2VwdGlvbiBhZnRlciAlZCB0cmllcyBpbiAlZiBzZWNcbiIgJSAoY291bnQsIHRpbWUyLXRp bWUxKSkKICAgICAgICAgICAgICAgIHJhaXNlCgogICAgZGVmIF9fZW50ZXJfXyhzZWxmKToKICAg ICAgICBzZWxmLmN1cnNvci5fX2VudGVyX18oKQogICAgICAgIHJldHVybiBzZWxmCgogICAgZGVm IF9fZXhpdF9fKHNlbGYsICpleGNpbmZvKToKICAgICAgICBzZWxmLmN1cnNvci5fX2V4aXRfXygq ZXhjaW5mbykKCiAgICBkZWYgX19nZXRpdGVtX18oc2VsZiwga2V5KToKICAgICAgICBkYXRhID0g c2VsZi5fZXhlY3V0ZSgiU0VMRUNUICogZnJvbSAlcyB3aGVyZSBrZXk9PzsiICUKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBzZWxmLnRhYmxlLCBba2V5XSkKICAgICAgICBmb3Igcm93IGlu IGRhdGE6CiAgICAgICAgICAgIHJldHVybiByb3dbMV0KICAgICAgICByYWlzZSBLZXlFcnJvcihr ZXkpCgogICAgZGVmIF9fZGVsaXRlbV9fKHNlbGYsIGtleSk6CiAgICAgICAgaWYga2V5IG5vdCBp biBzZWxmOgogICAgICAgICAgICByYWlzZSBLZXlFcnJvcihrZXkpCiAgICAgICAgc2VsZi5fZXhl Y3V0ZSgiREVMRVRFIGZyb20gJXMgd2hlcmUga2V5PT87IiAlIHNlbGYudGFibGUsIFtrZXldKQoK ICAgIGRlZiBfX3NldGl0ZW1fXyhzZWxmLCBrZXksIHZhbHVlKToKICAgICAgICBpZiBub3QgaXNp bnN0YW5jZShrZXksIGJhc2VzdHJpbmcpOgogICAgICAgICAgICByYWlzZSBUeXBlRXJyb3IoJ09u bHkgc3RyaW5nIGtleXMgYXJlIHN1cHBvcnRlZCcpCiAgICAgICAgZWxpZiBub3QgaXNpbnN0YW5j ZSh2YWx1ZSwgYmFzZXN0cmluZyk6CiAgICAgICAgICAgIHJhaXNlIFR5cGVFcnJvcignT25seSBz dHJpbmcgdmFsdWVzIGFyZSBzdXBwb3J0ZWQnKQoKICAgICAgICBkYXRhID0gc2VsZi5fZXhlY3V0 ZSgiU0VMRUNUICogZnJvbSAlcyB3aGVyZSBrZXk9PzsiICUKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBzZWxmLnRhYmxlLCBba2V5XSkKICAgICAgICBleGlzdHMgPSBsZW4obGlz dChkYXRhKSkKICAgICAgICBpZiBleGlzdHM6CiAgICAgICAgICAgIHNlbGYuX2V4ZWN1dGUoIlVQ REFURSAlcyBTRVQgdmFsdWU9PyBXSEVSRSBrZXk9PzsiICUgc2VsZi50YWJsZSwKICAgICAgICAg ICAgICAgICAgICAgICAgICBbdmFsdWUsIGtleV0pCiAgICAgICAgZWxzZToKICAgICAgICAgICAg c2VsZi5fZXhlY3V0ZSgiSU5TRVJUIGludG8gJXMoa2V5LCB2YWx1ZSkgdmFsdWVzICg/LCA/KTsi ICUKICAgICAgICAgICAgICAgICAgICAgICAgICBzZWxmLnRhYmxlLCBba2V5LCB2YWx1ZV0pCgog ICAgZGVmIF9fY29udGFpbnNfXyhzZWxmLCBrZXkpOgogICAgICAgIHJldHVybiBrZXkgaW4gc2V0 KHNlbGYpCgogICAgZGVmIF9fbGVuX18oc2VsZik6CiAgICAgICAgZGF0YSA9IHNlbGYuX2V4ZWN1 dGUoIlNFTEVDVCBDT1VOVChrZXkpIEZST00gJXM7IiAlIHNlbGYudGFibGUpCiAgICAgICAgZm9y IHJvdyBpbiBkYXRhOgogICAgICAgICAgICByZXR1cm4gcm93WzBdCgogICAgZGVmIF9faXRlcl9f KHNlbGYpOgogICAgICAgIGRhdGEgPSBzZWxmLl9leGVjdXRlKCJTRUxFQ1Qga2V5IEZST00gJXM7 IiAlIHNlbGYudGFibGUpCiAgICAgICAgcmV0dXJuIChyb3dbMF0gZm9yIHJvdyBpbiBkYXRhKQoK ICAgIGRlZiBfX2x0X18oc2VsZiwgb3RoZXIpOgogICAgICAgIGlmIG5vdCBpc2luc3RhbmNlKG90 aGVyLCBNYXBwaW5nKToKICAgICAgICAgICAgcmFpc2UgTm90SW1wbGVtZW50ZWQKCiAgICAgICAg cmV0dXJuIGxlbihzZWxmKSA8IGxlbihvdGhlcikKCiAgICBkZWYgdmFsdWVzKHNlbGYpOgogICAg ICAgIHJldHVybiBsaXN0KHNlbGYuaXRlcnZhbHVlcygpKQoKICAgIGRlZiBpdGVydmFsdWVzKHNl bGYpOgogICAgICAgIGRhdGEgPSBzZWxmLl9leGVjdXRlKCJTRUxFQ1QgdmFsdWUgRlJPTSAlczsi ICUgc2VsZi50YWJsZSkKICAgICAgICByZXR1cm4gKHJvd1swXSBmb3Igcm93IGluIGRhdGEpCgog ICAgZGVmIGl0ZW1zKHNlbGYpOgogICAgICAgIHJldHVybiBsaXN0KHNlbGYuaXRlcml0ZW1zKCkp CgogICAgZGVmIGl0ZXJpdGVtcyhzZWxmKToKICAgICAgICByZXR1cm4gc2VsZi5fZXhlY3V0ZSgi U0VMRUNUICogRlJPTSAlczsiICUgc2VsZi50YWJsZSkKCiAgICBkZWYgY2xlYXIoc2VsZik6CiAg ICAgICAgc2VsZi5fZXhlY3V0ZSgiREVMRVRFIEZST00gJXM7IiAlIHNlbGYudGFibGUpCgogICAg ZGVmIGhhc19rZXkoc2VsZiwga2V5KToKICAgICAgICByZXR1cm4ga2V5IGluIHNlbGYKCmRlZiBj b25uZWN0KGRhdGFiYXNlKToKICAgIHJldHVybiBzcWxpdGUzLmNvbm5lY3QoZGF0YWJhc2UsIHRp bWVvdXQ9MSwgaXNvbGF0aW9uX2xldmVsPU5vbmUpCgpkZWYgcGVyc2lzdChkb21haW4pOgoJY2Fj aGVmaWxlID0gb3MucGF0aC5qb2luKCIvdG1wIiwgIjE3NjEuc3FsaXRlMyIpCgljb25uZWN0aW9u ID0gY29ubmVjdChjYWNoZWZpbGUpCglyZXR1cm4gU1FMVGFibGUoY29ubmVjdGlvbiwgZG9tYWlu KQoKZGVmIGFjY2Vzcyhsb2NhbGNvdW50cyk6CglrZXkgPSBzdHIocmFuZG9tLnJhbmRyYW5nZSgw LDk5OTk5OTk5OTksMSkpCglsYXN0X3JldiA9IGxvY2FsY291bnRzLmdldChrZXkgKyAnX3Jldicp Cgljb3VudCA9IGxvY2FsY291bnRzLmdldChrZXkgKyAnX2NvdW50Jykgb3IgIjAiCgljb3VudCA9 IHN0cihpbnQoY291bnQpICsgMSkKCQoJbG9jYWxjb3VudHNba2V5ICsgJ19yZXYnXSA9IGtleQoJ bG9jYWxjb3VudHNba2V5ICsgJ19jb3VudCddID0gY291bnQKCmRlZiBkb190YXNrKHQpOgoJcHJp bnQoInRocmVhZCAlZCBzdGFydGVkXG4iICUgdCkKCWkgPSAwCgl0cnk6CgkJbG9jYWxjb3VudHMg PSBwZXJzaXN0KCdCQl9VUklfTE9DQUxDT1VOVCcpCgkJd2hpbGUgaSA8IDEwOgoJCQlhY2Nlc3Mo bG9jYWxjb3VudHMpCgkJCWkgPSBpICsgMQoJZXhjZXB0OgoJCXByaW50KCJmYWlsZWQgYXQgdGhy ZWFkICVkLCAlZCB0aW1lXG4iICUgKHQsaSkpCgkJaW1wb3J0IHRyYWNlYmFjawoJCXRyYWNlYmFj ay5wcmludF9leGMoNSkKCXByaW50ICgidGhyZWFkICVkIGZpbmlzaGVkXG4iICUgdCkKCmNhY2hl ZmlsZSA9IG9zLnBhdGguam9pbigiL3RtcCIsICIxNzYxLnNxbGl0ZTMiKQppZiBvcy5wYXRoLmV4 aXN0cyhjYWNoZWZpbGUpOgoJb3MucmVtb3ZlKGNhY2hlZmlsZSkKbnVtID0gMApwb29sID0gW10K d2hpbGUgbnVtIDwgODA6Cglwb29sLmFwcGVuZCh0aHJlYWRpbmcuVGhyZWFkKHRhcmdldD1kb190 YXNrLCBhcmdzPShudW0sKSkpCgludW0gPSBudW0gKzEKCmZvciBpIGluIHBvb2w6CglpLnN0YXJ0 KCkKCmZvciBpIGluIHBvb2w6CglpLmpvaW4oKQo=