Bug 5051 - ERROR: Can NOT get PRAUTO, exception timed out
Summary: ERROR: Can NOT get PRAUTO, exception timed out
Status: VERIFIED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High critical
Target Milestone: 1.5 M5
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2013-08-27 02:15 UTC by Jonathan Liu
Modified: 2013-10-17 14:46 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Liu 2013-08-27 02:15:50 UTC
After updating poky to "master:4517cdbfafdc7c5fa16bb5d09ed70c8e0da6a8c8", when building an image after a while I get:
ERROR: Can NOT get PRAUTO, exception timed out
ERROR: Function failed: package_get_auto_pr

It has happened three times so far. I am using the following relevant options in my local.conf:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
PRSERV_HOST = "localhost:0"

Target machine is i586 Linux.
Comment 1 Jonathan Liu 2013-08-27 02:25:55 UTC
The time of each run before it gave the error:
23m53.394s
2m41.952s
18m43.564s

I run bitbake with time nice -n19. I suspect it may be due to http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=bfab986ccd59636d0b0e79309f020e55481b0567.
Comment 2 Jonathan Liu 2013-08-27 05:06:53 UTC
The errors disappears if I revert bfab986ccd59636d0b0e79309f020e55481b0567.
Comment 3 Richard Purdie 2013-08-27 23:56:04 UTC
If you swap timeout=5 to timeout=30 in that commit are things any better?
Comment 4 Jonathan Liu 2013-08-28 02:48:43 UTC
It would probably help but I have already built everything without the patch. I need a reliable method to reproduce it first.
Comment 5 Jonathan Liu 2013-08-28 22:48:27 UTC
Having timeout of 30 seems to fix it for me.
Comment 6 Richard Purdie 2013-08-29 13:41:47 UTC
The performance of the PR service is drastically improved as of http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d5e860be4c12a9d6655665af391ffd6f9351f8eb so can you see if you still have issues after this change?
Comment 7 Jonathan Liu 2013-08-29 13:48:55 UTC
I still have same error after http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d5e860be4c12a9d6655665af391ffd6f9351f8eb with timeout of 5.
Comment 8 Jonathan Liu 2013-08-29 22:52:39 UTC
(In reply to comment #6)
> The performance of the PR service is drastically improved as of
> http://git.yoctoproject.org/cgit.cgi/poky/commit/
> ?id=d5e860be4c12a9d6655665af391ffd6f9351f8eb so can you see if you still
> have issues after this change?

It seems to be even worse now. After d5e860be4c12a9d6655665af391ffd6f9351f8eb using timeout of 30 fails with the following error (and more frequently):
ERROR: Can NOT get PRAUTO from remote PR service
ERROR: Function failed: package_get_auto_pr

build/cache/prserve.log contains:
ERROR: database table is locked: PRMAIN_nohist
NOTE: PRServer: stopping...
Comment 9 Richard Purdie 2013-08-30 16:54:48 UTC
I've pushed a change making the retry match for "table is locked" which it didn't previously. I've also increased the timeout to 60s since its causing people problems. If you could see if these two patches combined, I'd appreciate it, thanks!
Comment 10 Richard Purdie 2013-09-01 08:04:10 UTC
I found further problem and went into this more deeply. There is now a series of patches on the bitbake mailing list which should improve the performance of the PR Server once and for all and resolve this problem, even with a 5s timeout.
Comment 11 Jonathan Liu 2013-09-01 08:51:25 UTC
It works after applying the two patches for 30 second timeout and SQLite retry fix.
Comment 12 Richard Purdie 2013-09-05 15:11:56 UTC
I think the various bitbake changes to improve the PR server responsiveness have resolved this, ending with commit http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c7b342903208cfb4cefaa0ec339c6ddfcd590acc
Comment 13 Stoicescu Cornel 2013-10-17 14:46:36 UTC
Tested with "timeout = 5" and there ware no issues.

Tested on poky dora/master:faf8f8660fae26a14578738b642fdeb08bcb9d54