Bug 13716 - Master] SSTATE_MIRROR hangs if mirror site is down.
Summary: Master] SSTATE_MIRROR hangs if mirror site is down.
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.1
Hardware: x86 Multiple
: High normal
Target Milestone: 4.0 M1
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-12-19 16:26 UTC by Armin Kuster
Modified: 2021-11-13 16:42 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Kuster 2019-12-19 16:26:18 UTC
I have these variables defined:

SOURCE_MIRROR_URL ?= "http://downloads.yoctoproject.org/mirror/sources/"
SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/PATH"


BB_SIGNATURE_HANDLER = "OEEquivHash"


The mirrors are down so there is a very long ETA. 

Possible need some sort of timeout and quit?
Comment 1 Michael Halstead 2021-10-26 19:55:59 UTC
We are planning a downtime for the YP mirrors late this year to move data centers. If there isn't a code fix for this by then I'll find a way to ensure quick timeouts during the downtime. I need to test if a server that responds with 503 temporarily unavailable for all requests allows for quick timeouts.
Comment 2 Richard Purdie 2021-11-09 13:26:11 UTC
I've sent out a mirror addition for uninative which partially helps some of the issues experienced last time.
Comment 3 Richard Purdie 2021-11-09 13:30:46 UTC
I also did some failure analysis. Last time when this happened, the webserver for sstate.yoctoproject.org was alive but it's filesystem had gone which meant connections to the system were hung open for a long period.

If I change my /etc/hosts to add something like:

192.168.1.123   sstate.yoctoproject.org git.yoctoproject.org downloads.yoctoproject.org

i.e. point at an invalid address, then the connections fail and the build proceeds just fine, either finding things from mirrors or not but it doesn't hang.

I therefore think the downtime later in the year should be ok as long as we don't have a half operating webserver hanging connections. We do need to make sure sources are available on mirrors though and Michael and I are putting a plan in place for that, uninative is already covered.

Once we have the mirrors sorted, this bug can drop in priority and we can analyse connection timeouts to address the other issue with a non-responsive server.
Comment 4 Richard Purdie 2021-11-11 14:26:48 UTC
Fix for the hang:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2c46245f449d2716566ef668da0bf48f2109643a

We just need an additional sources mirror url and we should be able to resolve this.