Bug 13050

Summary: Eclipse autobuilder task timing out
Product: [Yocto Project Subprojects] Eclipse Plugin Reporter: Ross Burton <ross.burton>
Component: eclipse-pluginAssignee: Chin Huat Ang <chin.huat.ang>
Status: CLOSED WONTFIX QA Contact: Toaster default assignee <toaster>
Severity: normal    
Priority: Medium+ CC: akuster, richard.purdie, TicoTimo, tim.orling, yp.ep.watcher, yp.watcher
Version: unspecified   
Target Milestone: 2.7 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Output of setup.log with p2 debug traces enabled
none
Build statistics for neon-master branch
none
Build statistics for oxygen-master branch none

Description Ross Burton 2018-11-29 17:05:24 UTC
The eclipse job is timing out during the build and being killed.  This is happening repeatedly so it's not just a one-off:

https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/262
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/264
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/265
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/266

Also as we're downloading 150MB every time, maybe we should also check in the source mirror for that and ask Michael to put it there.
Comment 2 Chin Huat Ang 2018-12-06 07:11:04 UTC
The correct links to autobuilder are:

eclipse-plugin-oxygen builder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/262
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/264

eclipse-plugin-neon builder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/265
https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/266"

I suspect this is a download.eclipse.org timeout issue as reported in https://bugs.eclipse.org/bugs/show_bug.cgi?id=515596 although I'm not able to reproduce the issue using ~20 fresh builds.

For the coming 2.7 release, there is plan to migrate Eclipse to maven build which could leverage nexus caching for mitigating intermittent timeout issues when downloading upstream artifacts. For earlier releases (2.5/2.6), perhaps we can update the eclipse build scripts to retry the downloads, or update autobuilder helper to restart the build?
Comment 3 Tim Orling 2018-12-06 15:45:41 UTC
The downloads should all be coming from the full eclipse mirror [1] 

[1] http://downloads.yoctoproject.org/eclipse-full/

for oxygen:
[2] http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/tree/scripts/setup.sh?h=oxygen-master#n261

On review, neon is not using the full mirror:
[3] http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/tree/scripts/setup.sh?h=neon-master#n253

I will prepare a patch for neon, but we need to check that what we expect to be on the full mirror actually exists and that the workers have a good connection to the full mirror.
Comment 4 Tim Orling 2018-12-06 16:34:45 UTC
Patch for neon submitted [1] and pushed [2].

First build with patch applied built in a new record time of 6:58 [3].

We still need to run more builds to see if the intermittent problem has been fixed.

[1] https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000207.html
[2] http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=neon/thud&id=923b6a40bd7934cadf99918303aaeb931e90cf59
[3] https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/297
Comment 5 Tim Orling 2018-12-08 17:51:44 UTC
We can build multiple eclipse-plugin-oxygen jobs in a row with no trouble:
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/305
through
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/322

and also eclipse-plugin-neon:

https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/310
through
https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/322

But we still see the timeout under other conditions:

https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/324
https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/325

In order to debug this, we have got to figure out a way to more repeatably trigger the failure.
Comment 6 Chin Huat Ang 2018-12-12 01:59:13 UTC
I was able to turn on additional debug traces to dump P2 debug messages. Apparently running setup.sh without --upstream still downloads all Eclipse upstream dependencies from Eclipse mirrors, despite setup.sh is already using the eclipse full mirror on downloads.yoctoproject.org.
Comment 7 Chin Huat Ang 2018-12-12 02:57:11 UTC
Created attachment 4402 [details]
Output of setup.log with p2 debug traces enabled

146 out of 182 upstream artifacts are still being downloaded from upstream mirrors, for example:

[p2] Tue Dec 11 15:15:21 MYT 2018 - [Worker-1] Selected mirror for artifact http://downloads.yoctoproject.org/eclipse-full/releases/oxygen/201804111000/plugins/org.eclipse.cdt.build.gcc.ui_1.0.0.201802261533.jar.pack.gz: Mirror(http://mirrors.neusoft.edu.cn/eclipse/releases/oxygen/201804111000/,0,-1)
Comment 8 Chin Huat Ang 2018-12-12 03:00:35 UTC
I tried the following instructions using "-vmargs -Declipse.p2.mirrors=false" to disabled P2 mirrors, but no luck so far.

https://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#Avoiding_mirrors.2C_even_when_using_p2.mirrorsURL
Comment 9 Chin Huat Ang 2018-12-13 19:14:54 UTC
I was able to disable P2 mirrors by adding eclipse.p2.mirrors=false to config.ini, but not using command line.

The following patches should help to improve the intermittent timeout issues. The first patch provides the ability to enable P2 debug traces so that we can tell whether eclipse-full mirror or other Eclipse mirrors is used, and the second patch provides the ability to force downloading CDT etc. from https://downloads.yoctoproject.org/eclipse-full/ only :

  neon: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000216.html
  oxygen: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000218.html

The autobuilder should invoke setup.sh like this:

  P2_INSTALL_DEBUG=1 P2_MIRRORS=0 ./setup.sh

I also have patches ready for master.
Comment 10 Chin Huat Ang 2018-12-13 19:23:06 UTC
Created attachment 4406 [details]
Build statistics for neon-master branch

Interestingly, disabling Eclipse Neon mirrors improves the build time from 500s to sub 300s, despite the Eclipse wiki says it might slowdown the build:

  https://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#Avoiding_mirrors.2C_even_when_using_p2.mirrorsURL

Disabling Eclipse Oxygen mirrors doesn't seem to yield noticeable build time changes.
Comment 11 Chin Huat Ang 2018-12-13 19:24:04 UTC
Created attachment 4407 [details]
Build statistics for oxygen-master branch
Comment 12 Chin Huat Ang 2018-12-19 04:58:16 UTC
v2 eliminates the need to set environment variables so that we don't need to modify the autobuilder configuration:

neon: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000224.html
oxygen: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2018-December/000227.html
Comment 13 Chin Huat Ang 2019-01-11 09:52:13 UTC
neon-master v3, oxygen-master v3, master v2:

  https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000262.html
Comment 15 Chin Huat Ang 2019-01-25 01:36:30 UTC
(In reply to comment #14)
> https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/463

This build was started just an hour before the fixes were merged to eclipse-yocto neon-master branch:

  http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=neon-master&id=2ca509f4c13e142c322791b90efbad3130668151

Build 464 seemed to have the fix incorporated where the verbose build message did not contain message in Comment 7 which would indicate that the build is still fetching from random Eclipse mirrors.

The intermittent timeout issue used to occur between 2-50+ builds, mostly between 10-30 builds. Once we pass these marks we can close this for good.
Comment 16 Richard Purdie 2019-01-28 17:39:40 UTC
Another timed out build: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/475
Comment 18 Armin Kuster 2019-05-23 14:53:44 UTC
we dropped eclipse from the AB so this is obsolete.
Comment 19 Armin Kuster 2019-05-23 14:53:57 UTC
closing