Bug 14767

Summary:	buildbot timeout of build for unknown reasons (fedora35-ty-2, genericx86-alt)
Product:	[Infrastructure] AutoBuilder	Reporter:	Richard Purdie <richard.purdie>
Component:	autobuilder	Assignee:	Unassigned <unassigned>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium	CC:	alexandre.belloni, infras.ab.watcher, Infras.watcher, pidge, randy.macleod
Version:	4.1
Target Milestone:	4.1
Hardware:	x86
OS:	Multiple
Whiteboard:	AB-INT
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Richard Purdie 2022-03-23 22:25:11 UTC

The buildnot UI reports the build timed out with no output but the build looks like it completed.
 
Worker: fedora35-ty-2
genericx86-alt target

https://autobuilder.yoctoproject.org/typhoon/#/builders/103/builds/3818

We can see the step timing information here:

https://autobuilder.yoctoproject.org/typhoon/api/v2/builders/103/builds/3818/steps/11

Translating those epoch values gives:

22 March 2022 17:21:42
22 March 2022 23:01:16

Logs saved here:

https://autobuilder.yocto.io/pub/failed-builds-data/fedora35-genericx86-alt-20220322/

With the endtime on https://autobuilder.yocto.io/pub/failed-builds-data/fedora35-genericx86-alt-20220322/bitbake-cookerdaemon.log showing a time about 10 mins after the start and a clear server exit.

This suggests the server really did complete and the UI hung at exit for some reason. Why? Don't know :(

Comment 1 Richard Purdie 2022-03-23 23:30:47 UTC

I am suspicious of 1785558 14:31:16.523180 More than one thread left?: [<_MainThread(MainThread, started 139817764251456)>, <Thread(asyncio_0, started 139817658435136)>]

Comment 2 Richard Purdie 2022-03-26 17:58:21 UTC

https://git.yoctoproject.org/poky/commit/?id=88fcf8422dd04372d73aacfef0ee50de09ef41bd and https://git.yoctoproject.org/poky/commit/?id=cff6c1a18d56e20d99d2331704dd490940c386e3 should handle the asyncio thread issue which is python 3.10 specific. Not sure if this will help the hang.

Comment 3 Richard Purdie 2022-06-16 15:05:52 UTC

Possibly fixed by https://git.yoctoproject.org/poky/commit/bitbake?id=5940020cfb721d79111072b3cd5d581fff59561b (there was a similar bug I can't find right now)