The eclipse job is timing out during the build and being killed. This is happening repeatedly so it's not just a one-off: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/262 https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/264 https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/265 https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/266 Also as we're downloading 150MB every time, maybe we should also check in the source mirror for that and ask Michael to put it there.
https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/289
The correct links to autobuilder are: eclipse-plugin-oxygen builder: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/262 https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/264 eclipse-plugin-neon builder: https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/265 https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/266" I suspect this is a download.eclipse.org timeout issue as reported in https://bugs.eclipse.org/bugs/show_bug.cgi?id=515596 although I'm not able to reproduce the issue using ~20 fresh builds. For the coming 2.7 release, there is plan to migrate Eclipse to maven build which could leverage nexus caching for mitigating intermittent timeout issues when downloading upstream artifacts. For earlier releases (2.5/2.6), perhaps we can update the eclipse build scripts to retry the downloads, or update autobuilder helper to restart the build?
The downloads should all be coming from the full eclipse mirror [1] [1] http://downloads.yoctoproject.org/eclipse-full/ for oxygen: [2] http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/tree/scripts/setup.sh?h=oxygen-master#n261 On review, neon is not using the full mirror: [3] http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/tree/scripts/setup.sh?h=neon-master#n253 I will prepare a patch for neon, but we need to check that what we expect to be on the full mirror actually exists and that the workers have a good connection to the full mirror.
Patch for neon submitted [1] and pushed [2]. First build with patch applied built in a new record time of 6:58 [3]. We still need to run more builds to see if the intermittent problem has been fixed. [1] https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000207.html [2] http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=neon/thud&id=923b6a40bd7934cadf99918303aaeb931e90cf59 [3] https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/297
We can build multiple eclipse-plugin-oxygen jobs in a row with no trouble: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/305 through https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/322 and also eclipse-plugin-neon: https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/310 through https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/322 But we still see the timeout under other conditions: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/324 https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/325 In order to debug this, we have got to figure out a way to more repeatably trigger the failure.
I was able to turn on additional debug traces to dump P2 debug messages. Apparently running setup.sh without --upstream still downloads all Eclipse upstream dependencies from Eclipse mirrors, despite setup.sh is already using the eclipse full mirror on downloads.yoctoproject.org.
Created attachment 4402 [details] Output of setup.log with p2 debug traces enabled 146 out of 182 upstream artifacts are still being downloaded from upstream mirrors, for example: [p2] Tue Dec 11 15:15:21 MYT 2018 - [Worker-1] Selected mirror for artifact http://downloads.yoctoproject.org/eclipse-full/releases/oxygen/201804111000/plugins/org.eclipse.cdt.build.gcc.ui_1.0.0.201802261533.jar.pack.gz: Mirror(http://mirrors.neusoft.edu.cn/eclipse/releases/oxygen/201804111000/,0,-1)
I tried the following instructions using "-vmargs -Declipse.p2.mirrors=false" to disabled P2 mirrors, but no luck so far. https://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#Avoiding_mirrors.2C_even_when_using_p2.mirrorsURL
I was able to disable P2 mirrors by adding eclipse.p2.mirrors=false to config.ini, but not using command line. The following patches should help to improve the intermittent timeout issues. The first patch provides the ability to enable P2 debug traces so that we can tell whether eclipse-full mirror or other Eclipse mirrors is used, and the second patch provides the ability to force downloading CDT etc. from https://downloads.yoctoproject.org/eclipse-full/ only : neon: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000216.html oxygen: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000218.html The autobuilder should invoke setup.sh like this: P2_INSTALL_DEBUG=1 P2_MIRRORS=0 ./setup.sh I also have patches ready for master.
Created attachment 4406 [details] Build statistics for neon-master branch Interestingly, disabling Eclipse Neon mirrors improves the build time from 500s to sub 300s, despite the Eclipse wiki says it might slowdown the build: https://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#Avoiding_mirrors.2C_even_when_using_p2.mirrorsURL Disabling Eclipse Oxygen mirrors doesn't seem to yield noticeable build time changes.
Created attachment 4407 [details] Build statistics for oxygen-master branch
v2 eliminates the need to set environment variables so that we don't need to modify the autobuilder configuration: neon: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000224.html oxygen: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000227.html
neon-master v3, oxygen-master v3, master v2: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000262.html
https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/463
(In reply to comment #14) > https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/463 This build was started just an hour before the fixes were merged to eclipse-yocto neon-master branch: http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=neon-master&id=2ca509f4c13e142c322791b90efbad3130668151 Build 464 seemed to have the fix incorporated where the verbose build message did not contain message in Comment 7 which would indicate that the build is still fetching from random Eclipse mirrors. The intermittent timeout issue used to occur between 2-50+ builds, mostly between 10-30 builds. Once we pass these marks we can close this for good.
Another timed out build: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/475
Still happening in both neon and oxygen: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/627 https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/622
we dropped eclipse from the AB so this is obsolete.
closing