Bug 14328

Summary:	Mitigate AB ptest intermittent failure
Product:	[QA/Testing] Package Testing (ptest)	Reporter:	Yi Fan Yu <yf3yu>
Component:	ptest	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	alexandre.belloni, randy.macleod
Version:	unspecified
Target Milestone:	3.4 M1
Hardware:	x86
OS:	Multiple
Whiteboard:	AB-INT
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know
Bug Depends on:
Bug Blocks:	14163

Description Yi Fan Yu 2021-03-25 15:35:02 UTC

It feels like an I/O load issue, work on some sort of `flock`/`dd` timeout is currently done to try to get to the bottom of the cause and verify if it is indeed I/O related.


2 solutions have been suggested in the Mar 25 triage 

IIRC they were: 

* cgroups control to give qemu an advantage over builds
* nice levels to make qemu higher priority (lowering the builds priority)

Comment 1 Randy MacLeod 2021-04-01 14:35:24 UTC

We were doing this anyway but this bug can be used to track things.

Comment 2 Richard Purdie 2021-04-15 13:35:10 UTC

Most autobuilder workers only have cgroups v1 which doesn't give the IO controls we'd need. We already set ionice/nice levels. 

The ionice level doesn't affect async writes.

We've now added support for running qemu images within tmpfs.

Comment 3 Alexandre Belloni 2021-04-15 14:58:58 UTC

Following the triage call, it seems necessary to:
 - timestamp the start and end of the image copy to tmpfs
 - double the qemu start timeout

Comment 4 Randy MacLeod 2021-05-20 15:13:32 UTC

Many changes including:
 - moving the qemu rootfs to tmpfs
 - limiting glibc self-test memory usage
 - adding smp to qemu image for x86, arm and changing the x86 target machine.
and likely more have improve ptests. Closing this generic bug. Open specific issues for additional problems.

Comment 5 Randy MacLeod 2021-05-20 15:13:45 UTC

As per previous comment.