Bug 5051

Summary:	ERROR: Can NOT get PRAUTO, exception timed out
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Jonathan Liu <net147>
Component:	bitbake	Assignee:	Richard Purdie <richard.purdie>
Status:	VERIFIED FIXED	QA Contact:
Severity:	critical
Priority:	High	CC:	corneliux.stoicescu, poky.bs.watcher, poky.watcher, sgw
Version:	unspecified
Target Milestone:	1.5 M5
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Jonathan Liu 2013-08-27 02:15:50 UTC

After updating poky to "master:4517cdbfafdc7c5fa16bb5d09ed70c8e0da6a8c8", when building an image after a while I get:
ERROR: Can NOT get PRAUTO, exception timed out
ERROR: Function failed: package_get_auto_pr

It has happened three times so far. I am using the following relevant options in my local.conf:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
PRSERV_HOST = "localhost:0"

Target machine is i586 Linux.

Comment 1 Jonathan Liu 2013-08-27 02:25:55 UTC

The time of each run before it gave the error:
23m53.394s
2m41.952s
18m43.564s

I run bitbake with time nice -n19. I suspect it may be due to http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=bfab986ccd59636d0b0e79309f020e55481b0567.

Comment 2 Jonathan Liu 2013-08-27 05:06:53 UTC

The errors disappears if I revert bfab986ccd59636d0b0e79309f020e55481b0567.

Comment 3 Richard Purdie 2013-08-27 23:56:04 UTC

If you swap timeout=5 to timeout=30 in that commit are things any better?

Comment 4 Jonathan Liu 2013-08-28 02:48:43 UTC

It would probably help but I have already built everything without the patch. I need a reliable method to reproduce it first.

Comment 5 Jonathan Liu 2013-08-28 22:48:27 UTC

Having timeout of 30 seems to fix it for me.

Comment 6 Richard Purdie 2013-08-29 13:41:47 UTC

The performance of the PR service is drastically improved as of http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d5e860be4c12a9d6655665af391ffd6f9351f8eb so can you see if you still have issues after this change?

Comment 7 Jonathan Liu 2013-08-29 13:48:55 UTC

I still have same error after http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d5e860be4c12a9d6655665af391ffd6f9351f8eb with timeout of 5.

Comment 8 Jonathan Liu 2013-08-29 22:52:39 UTC

(In reply to comment #6)
> The performance of the PR service is drastically improved as of
> http://git.yoctoproject.org/cgit.cgi/poky/commit/
> ?id=d5e860be4c12a9d6655665af391ffd6f9351f8eb so can you see if you still
> have issues after this change?

It seems to be even worse now. After d5e860be4c12a9d6655665af391ffd6f9351f8eb using timeout of 30 fails with the following error (and more frequently):
ERROR: Can NOT get PRAUTO from remote PR service
ERROR: Function failed: package_get_auto_pr

build/cache/prserve.log contains:
ERROR: database table is locked: PRMAIN_nohist
NOTE: PRServer: stopping...

Comment 9 Richard Purdie 2013-08-30 16:54:48 UTC

I've pushed a change making the retry match for "table is locked" which it didn't previously. I've also increased the timeout to 60s since its causing people problems. If you could see if these two patches combined, I'd appreciate it, thanks!

Comment 10 Richard Purdie 2013-09-01 08:04:10 UTC

I found further problem and went into this more deeply. There is now a series of patches on the bitbake mailing list which should improve the performance of the PR Server once and for all and resolve this problem, even with a 5s timeout.

Comment 11 Jonathan Liu 2013-09-01 08:51:25 UTC

It works after applying the two patches for 30 second timeout and SQLite retry fix.

Comment 12 Richard Purdie 2013-09-05 15:11:56 UTC

I think the various bitbake changes to improve the PR server responsiveness have resolved this, ending with commit http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c7b342903208cfb4cefaa0ec339c6ddfcd590acc

Comment 13 Stoicescu Cornel 2013-10-17 14:46:36 UTC

Tested with "timeout = 5" and there ware no issues.

Tested on poky dora/master:faf8f8660fae26a14578738b642fdeb08bcb9d54