Bug 10589

Summary:	Bugzilla can't be reliably accessed at ~08:45 UTC each day
Product:	[Infrastructure] Bugzilla	Reporter:	Joshua Lock <joshuagloe>
Component:	bugzilla	Assignee:	Michael Halstead <mhalstead>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	infras.bug.watcher, Infras.watcher, randy.macleod, richard.purdie
Version:	unspecified
Target Milestone:	Q4
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Joshua Lock 2016-11-03 09:50:27 UTC

Around 08:45 GMT/UTC each day we reliably run into errors accessing the Yocto Project Bugzilla.

The errors are error 28 from the storage engine, i.e.

"undef error - DBD::mysql::st execute failed: Got error 28 from storage engine...."

When I first saw this Michael suggested that this error seems to occur when the bugzilla database is being backed up and that we might want a new backup strategy.

As an interim solution it could be useful to move the backup earlier in the day, 08:45 UTC is when several of our contributors and users in Europe are already trying to access Bugzilla.

Comment 1 Joshua Lock 2016-11-04 09:11:22 UTC

FWIW the window of unavailability is quite long. I was unable to access bugzilla from ~08:45 - ~09:10

Comment 2 Michael Halstead 2016-11-05 03:40:19 UTC

I've altered the backup strategy. Bugzilla now stays online during the backup process. I've also moved it to 5AM UTC where the performance hit will trouble fewer users.