| Summary: | eclipse oxygen-master build failing consistently | ||
|---|---|---|---|
| Product: | [Infrastructure] Infrastructure General | Reporter: | Anuj Mittal <anuj.mittal> |
| Component: | infrastructure-general | Assignee: | Tim Orling <tim.orling> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | akuster, chin.huat.ang, mhalstead, tim.orling, yp.ep.watcher, yp.watcher |
| Version: | 2.7 | ||
| Target Milestone: | 2.7 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
| Bug Depends on: | |||
| Bug Blocks: | 13105 | ||
|
Description
Anuj Mittal
2018-12-24 06:56:05 UTC
Something's wrong with the Eclipse mirror on yoctoproject.org: --2018-12-26 10:18:21-- http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz Resolving downloads.yoctoproject.org (downloads.yoctoproject.org)... 198.145.29.63 Connecting to downloads.yoctoproject.org (downloads.yoctoproject.org)|198.145.29.63|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2018-12-26 10:18:22 ERROR 404: Not Found. Our mirror is up to date. Here is the listing for the drops4 directory upstream which matches ours: drwxrwsr-x 4,096 2018/09/26 06:39:45 P20180926-0920 drwxrwsr-x 4,096 2018/09/30 07:19:54 P20180930-1000 drwxrwsr-x 4,096 2018/09/30 23:33:51 P20181001-0215 drwxrwsr-x 4,096 2018/10/01 02:47:49 P20181001-0525 drwxrwsr-x 4,096 2018/10/03 02:19:21 P20181003-0450 drwxrwsr-x 4,096 2018/10/03 07:34:34 P20181003-1015 drwxrwsr-x 4,096 2018/10/08 03:10:59 P20181008-0545 drwxrwsr-x 4,096 2018/10/10 05:03:16 P20181010-0735 drwxrwsr-x 4,096 2018/10/10 05:44:28 P20181010-0815 drwxrwsr-x 4,096 2018/12/19 07:04:40 R-4.10-201812060815 drwxrwsr-x 4,096 2018/06/27 07:13:04 R-4.8-201806110500 drwxrwsr-x 4,096 2018/09/19 06:51:27 R-4.9-201809060745 R-4.7.3a-201803300640 appears to have been removed. In addition to the eclipse-full mirror we have a Yocto Project specific mirror which does not track upstream at http://downloads.yoctoproject.org/eclipse/downloads/drops4/. This only contains files that have been requested and they are not removed. I think we need a way to determine which files from the full mirror should be permanently archived and to update these scripts to point at the permanent location. Who can advise? R-4.7.3a-201803300640 has been moved to: https://archive.eclipse.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/ It looks like downloads.eclipse.org only contains the latest 3 releases (4.8 to 4.10), while archive.eclipse.org contains every known releases since 2001 (1.0 to 4.9), except the latest release. Can we also mirror archive.eclipse.org? If this is too much then perhaps we can limit to anything newer than 4.6.3? oxygen-4.7.2 is still available on our old eclipse partial mirror, but not the 4.7.3 or 4.7.3a versions. One option would be to mirror 4.7.3a [1] from archive (might as well do 4.7.3 [2] while we are at it) to our old eclipse partial mirror. [1] https://archive.eclipse.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/ [2] https://archive.eclipse.org/eclipse/downloads/drops4/R-4.7.3-201803010715/ Otherwise, we will have to revert to downloading from upstream, which is slow and has also caused build failures (timeouts). Ultimately, this begins to spell the end of support for Oxygen. We have already seen the same happen with Neon. Upstream has had three releases since Oxygen 4.7.3a (Photon, 2018-09 and 2018-12). found 13050, will it help ? I've downloaded the x86_64 SDK files from the archive and put them up at the following: http://downloads.yoctoproject.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz and http://downloads.yoctoproject.org/eclipse/downloads/drops4/R-4.7.3-201803010715/eclipse-SDK-4.7.3-linux-gtk-x86_64.tar.gz The archive is not available for mirroring at this time so I've only put the single files in place instead of the full directory contents. I've also temporarily disabled deletions on the full mirror and put http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz back in place. I hope this will unblock while we figure out a permanent solution. After getting a green build I've restored proper mirror functionality and allowed http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz to be removed. I've also added a redirect to http://downloads.yoctoproject.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz. If this also produces green builds 2.5.2 is fine as we can leave this in place. If this doesn't work we should wait for a patch to update the download location before releasing. The build uses wget with redirection enabled so I think it will work for all normal use cases. got a clean thud build. I will launch sumo just to check. The most simple solution I can think of is to automatically copy everything from http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4 to http://downloads.yoctoproject.org/eclipse/downloads/drops4 after each mirror sync. It seems the only files we care about are in that directory. Then we should only point at the partial mirror at http://downloads.yoctoproject.org/eclipse/downloads/drops4 going forward. Aside from growing over time and keeping files we don't need around are there any issues with that plan? These files aren't super large (currently 9GB) so we should be okay to keep them needed or not. (In reply to comment #6) > found 13050, will it help ? This is a separate issue which happens earlier than 13050, as the Eclipse build script failed to download the missing Eclipse archive. Once we figure out this, we should revisit 13050 where the build script is seeing intermittent timeout issue when downloading additional features. (In reply to comment #11) > The most simple solution I can think of is to automatically copy everything > from http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4 > to http://downloads.yoctoproject.org/eclipse/downloads/drops4 after each > mirror sync. It seems the only files we care about are in that directory. > Then we should only point at the partial mirror at > http://downloads.yoctoproject.org/eclipse/downloads/drops4 going forward. > How about mirror both: https://download.eclipse.org/eclipse/downloads/drops4/ https://archive.eclipse.org/eclipse/downloads/drops4/ ... to: http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4 http://downloads.yoctoproject.org/eclipse/eclipse/downloads/drops4 ... respectively. This way the the mirrors on yoctoproject.org would be near identical to *.eclipse.org. In the future whenever upstream moves resources from download.eclipse.org to archive.eclipse.org, we'll just need to patch eclipse-yocto build script to switch from eclipse-full/ to eclipse/ on yoctoproject.org? The old approach was to manually copy what was needed to the partial mirror. We tried to get away from the manual nature of that (and the sensitivity to upstream adding new releases) by using the full mirror. This incident shows that we still need to keep track of the items that get archived by upstream. We had changed the setup script to pull the SDK from our full mirror recently, because we saw some timeout issues (as Chin Huat mentioned, not directly related to 13050). However, the builds would still have failed because upstream moved the SDKs for oxygen to archive. Michael: If disk space is no concern, we can periodically automatically backup the full mirror to our partial mirror (or as you suggest disable deletions). I'd rather not do that for EVERYTHING, but rather on what we know we need to back up. The SDKs. These are the urls like .../drops4/R-<version>-<timestamp>/R-<version>-<timestamp>-linux-gtk-x86_64.tar.gz. The R stands for Release. In the mean time, can either Chin Huat or Chee Yang prepare a patch to update the oxygen-master setup.sh script to pull from archive.eclipse.org when the --upstream flag is set and from the http://downloads.yoctoproject.org/eclipse/downloads/drops4/{R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz} URL by default. We do not want to rely on the redirect. There is also apparently no good time for vacation ;) Patch to download Eclipse Oxygen from archive.eclipse.org: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000237.html All files matching *linux-gtk-x86_64.tar.gz from the full mirror drops4/R-* directories are copied to http://downloads.yoctoproject.org/eclipse/downloads/drops4/ now and will be updated hours. Files won't be removed from that location but WILL be removed from the eclipse-full location when upstream moves them to archive. All Yocto Project code should point at files in http://downloads.yoctoproject.org/eclipse/downloads/drops4/ going forward. If needed files are missing from this permanent location please raise an issue. Will this work? (In reply to comment #17) ... > Will this work? +1 from me. Tim has merged the patch resolving this issue as far as I know. Correct: http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=oxygen-master&id=f64e2007ecc893243e011f3d4b1bba3b2ceda428 and http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=oxygen-master&id=a76c846ca3e4fef3bcfe6c7b6c802f727d0e9314 From: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000239.html and https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000240.html not sure if this is related. eclipse on sumo failed this time. passed on previous build. command timed out: 16200 seconds without output running [b'/home/pokybuild/yocto-worker/eclipse-plugin-neon/yocto-autobuilder-helper/scripts/run-config', b'eclipse-plugin-neon', b'/home/pokybuild/yocto-worker/eclipse-plugin-neon/build/build', b'stable/sumo-nmut', b'git://git.yoctoproject.org/poky-contrib', b'-s', b'', b'-b', b'', b'-p', b'', b'-u', b'https://autobuilder.yoctoproject.org/typhoon/#builders/29/builds/481', b'-r', b'/srv/autobuilder/autobuilder.yoctoproject.org/pub/non-release/20190128-11/testresults', b'-q'], attempting to kill process killed by signal 9 https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/481 |