13101 2018-12-24 06:56:05 +0000 eclipse oxygen-master build failing consistently 2019-01-28 21:03:18 +0000 1 1 5 Infrastructure Infrastructure General infrastructure-general 2.7 x86 Multiple RESOLVED FIXED Medium+ normal 2.7 M2 13105 1 anuj.mittal tim.orling akuster chin.huat.ang mhalstead tim.orling yp.ep.watcher yp.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 82590 0 anuj.mittal 2018-12-24 06:56:05 +0000 ==================================================================================================== Step 1/1: Running 'plain' command cd scripts; ./setup.sh; ECLIPSE_HOME=`pwd`/eclipse/ ./build.sh oxygen-master master development (1545613400.3: 0.0) ==================================================================================================== Running 'cd scripts; ./setup.sh; ECLIPSE_HOME=`pwd`/eclipse/ ./build.sh oxygen-master master development' with output to /home/pokybuild/yocto-worker/eclipse-plugin-oxygen/build/build/command.log.1d ==================================================================================================== ERROR: Command cd scripts; ./setup.sh; ECLIPSE_HOME=`pwd`/eclipse/ ./build.sh oxygen-master master development failed with exit code 1, see errors above. (1545613400.5: 0.2) ==================================================================================================== Please see: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/371/steps/7/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/370/steps/7/logs/stdio And also on thud: https://autobuilder.yoctoproject.org/typhoon/#/builders/25/builds/369/steps/7/logs/stdio 82602 1 chin.huat.ang 2018-12-26 02:29:22 +0000 Something's wrong with the Eclipse mirror on yoctoproject.org: --2018-12-26 10:18:21-- http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz Resolving downloads.yoctoproject.org (downloads.yoctoproject.org)... 198.145.29.63 Connecting to downloads.yoctoproject.org (downloads.yoctoproject.org)|198.145.29.63|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2018-12-26 10:18:22 ERROR 404: Not Found. 82605 2 mhalstead 2018-12-26 23:11:59 +0000 Our mirror is up to date. Here is the listing for the drops4 directory upstream which matches ours: drwxrwsr-x 4,096 2018/09/26 06:39:45 P20180926-0920 drwxrwsr-x 4,096 2018/09/30 07:19:54 P20180930-1000 drwxrwsr-x 4,096 2018/09/30 23:33:51 P20181001-0215 drwxrwsr-x 4,096 2018/10/01 02:47:49 P20181001-0525 drwxrwsr-x 4,096 2018/10/03 02:19:21 P20181003-0450 drwxrwsr-x 4,096 2018/10/03 07:34:34 P20181003-1015 drwxrwsr-x 4,096 2018/10/08 03:10:59 P20181008-0545 drwxrwsr-x 4,096 2018/10/10 05:03:16 P20181010-0735 drwxrwsr-x 4,096 2018/10/10 05:44:28 P20181010-0815 drwxrwsr-x 4,096 2018/12/19 07:04:40 R-4.10-201812060815 drwxrwsr-x 4,096 2018/06/27 07:13:04 R-4.8-201806110500 drwxrwsr-x 4,096 2018/09/19 06:51:27 R-4.9-201809060745 R-4.7.3a-201803300640 appears to have been removed. In addition to the eclipse-full mirror we have a Yocto Project specific mirror which does not track upstream at http://downloads.yoctoproject.org/eclipse/downloads/drops4/. This only contains files that have been requested and they are not removed. I think we need a way to determine which files from the full mirror should be permanently archived and to update these scripts to point at the permanent location. Who can advise? 82608 3 chin.huat.ang 2018-12-27 02:28:40 +0000 R-4.7.3a-201803300640 has been moved to: https://archive.eclipse.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/ It looks like downloads.eclipse.org only contains the latest 3 releases (4.8 to 4.10), while archive.eclipse.org contains every known releases since 2001 (1.0 to 4.9), except the latest release. Can we also mirror archive.eclipse.org? If this is too much then perhaps we can limit to anything newer than 4.6.3? 82610 4 tim.orling 2018-12-27 18:38:11 +0000 oxygen-4.7.2 is still available on our old eclipse partial mirror, but not the 4.7.3 or 4.7.3a versions. One option would be to mirror 4.7.3a [1] from archive (might as well do 4.7.3 [2] while we are at it) to our old eclipse partial mirror. [1] https://archive.eclipse.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/ [2] https://archive.eclipse.org/eclipse/downloads/drops4/R-4.7.3-201803010715/ Otherwise, we will have to revert to downloading from upstream, which is slow and has also caused build failures (timeouts). 82611 5 tim.orling 2018-12-27 18:40:58 +0000 Ultimately, this begins to spell the end of support for Oxygen. We have already seen the same happen with Neon. Upstream has had three releases since Oxygen 4.7.3a (Photon, 2018-09 and 2018-12). 82651 6 akuster 2019-01-03 16:10:53 +0000 found 13050, will it help ? 82654 7 mhalstead 2019-01-03 16:25:55 +0000 I've downloaded the x86_64 SDK files from the archive and put them up at the following: http://downloads.yoctoproject.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz and http://downloads.yoctoproject.org/eclipse/downloads/drops4/R-4.7.3-201803010715/eclipse-SDK-4.7.3-linux-gtk-x86_64.tar.gz The archive is not available for mirroring at this time so I've only put the single files in place instead of the full directory contents. I've also temporarily disabled deletions on the full mirror and put http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz back in place. I hope this will unblock while we figure out a permanent solution. 82664 8 mhalstead 2019-01-03 21:46:34 +0000 After getting a green build I've restored proper mirror functionality and allowed http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz to be removed. I've also added a redirect to http://downloads.yoctoproject.org/eclipse/downloads/drops4/R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz. If this also produces green builds 2.5.2 is fine as we can leave this in place. If this doesn't work we should wait for a patch to update the download location before releasing. 82669 9 mhalstead 2019-01-03 23:40:07 +0000 The build uses wget with redirection enabled so I think it will work for all normal use cases. 82671 10 akuster 2019-01-04 00:06:00 +0000 got a clean thud build. I will launch sumo just to check. 82672 11 mhalstead 2019-01-04 00:15:54 +0000 The most simple solution I can think of is to automatically copy everything from http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4 to http://downloads.yoctoproject.org/eclipse/downloads/drops4 after each mirror sync. It seems the only files we care about are in that directory. Then we should only point at the partial mirror at http://downloads.yoctoproject.org/eclipse/downloads/drops4 going forward. Aside from growing over time and keeping files we don't need around are there any issues with that plan? These files aren't super large (currently 9GB) so we should be okay to keep them needed or not. 82673 12 chin.huat.ang 2019-01-04 02:19:27 +0000 (In reply to comment #6) > found 13050, will it help ? This is a separate issue which happens earlier than 13050, as the Eclipse build script failed to download the missing Eclipse archive. Once we figure out this, we should revisit 13050 where the build script is seeing intermittent timeout issue when downloading additional features. 82674 13 chin.huat.ang 2019-01-04 03:03:09 +0000 (In reply to comment #11) > The most simple solution I can think of is to automatically copy everything > from http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4 > to http://downloads.yoctoproject.org/eclipse/downloads/drops4 after each > mirror sync. It seems the only files we care about are in that directory. > Then we should only point at the partial mirror at > http://downloads.yoctoproject.org/eclipse/downloads/drops4 going forward. > How about mirror both: https://download.eclipse.org/eclipse/downloads/drops4/ https://archive.eclipse.org/eclipse/downloads/drops4/ ... to: http://downloads.yoctoproject.org/eclipse-full/eclipse/downloads/drops4 http://downloads.yoctoproject.org/eclipse/eclipse/downloads/drops4 ... respectively. This way the the mirrors on yoctoproject.org would be near identical to *.eclipse.org. In the future whenever upstream moves resources from download.eclipse.org to archive.eclipse.org, we'll just need to patch eclipse-yocto build script to switch from eclipse-full/ to eclipse/ on yoctoproject.org? 82720 14 tim.orling 2019-01-10 02:27:07 +0000 The old approach was to manually copy what was needed to the partial mirror. We tried to get away from the manual nature of that (and the sensitivity to upstream adding new releases) by using the full mirror. This incident shows that we still need to keep track of the items that get archived by upstream. We had changed the setup script to pull the SDK from our full mirror recently, because we saw some timeout issues (as Chin Huat mentioned, not directly related to 13050). However, the builds would still have failed because upstream moved the SDKs for oxygen to archive. Michael: If disk space is no concern, we can periodically automatically backup the full mirror to our partial mirror (or as you suggest disable deletions). I'd rather not do that for EVERYTHING, but rather on what we know we need to back up. The SDKs. These are the urls like .../drops4/R-<version>-<timestamp>/R-<version>-<timestamp>-linux-gtk-x86_64.tar.gz. The R stands for Release. In the mean time, can either Chin Huat or Chee Yang prepare a patch to update the oxygen-master setup.sh script to pull from archive.eclipse.org when the --upstream flag is set and from the http://downloads.yoctoproject.org/eclipse/downloads/drops4/{R-4.7.3a-201803300640/eclipse-SDK-4.7.3a-linux-gtk-x86_64.tar.gz} URL by default. We do not want to rely on the redirect. 82721 15 tim.orling 2019-01-10 02:27:59 +0000 There is also apparently no good time for vacation ;) 82744 16 chin.huat.ang 2019-01-10 23:15:57 +0000 Patch to download Eclipse Oxygen from archive.eclipse.org: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000237.html 82745 17 mhalstead 2019-01-11 00:25:05 +0000 All files matching *linux-gtk-x86_64.tar.gz from the full mirror drops4/R-* directories are copied to http://downloads.yoctoproject.org/eclipse/downloads/drops4/ now and will be updated hours. Files won't be removed from that location but WILL be removed from the eclipse-full location when upstream moves them to archive. All Yocto Project code should point at files in http://downloads.yoctoproject.org/eclipse/downloads/drops4/ going forward. If needed files are missing from this permanent location please raise an issue. Will this work? 82746 18 chin.huat.ang 2019-01-11 00:45:58 +0000 v2: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000239.html 82749 19 chin.huat.ang 2019-01-11 09:50:58 +0000 (In reply to comment #17) ... > Will this work? +1 from me. 82757 20 mhalstead 2019-01-11 18:11:11 +0000 Tim has merged the patch resolving this issue as far as I know. 82759 21 tim.orling 2019-01-11 18:33:59 +0000 Correct: http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=oxygen-master&id=f64e2007ecc893243e011f3d4b1bba3b2ceda428 and http://git.yoctoproject.org/cgit/cgit.cgi/eclipse-yocto/commit/?h=oxygen-master&id=a76c846ca3e4fef3bcfe6c7b6c802f727d0e9314 From: https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000239.html and https://lists.yoctoproject.org/pipermail/eclipse-yocto/2019-January/000240.html 82884 22 akuster 2019-01-28 21:03:18 +0000 not sure if this is related. eclipse on sumo failed this time. passed on previous build. command timed out: 16200 seconds without output running [b'/home/pokybuild/yocto-worker/eclipse-plugin-neon/yocto-autobuilder-helper/scripts/run-config', b'eclipse-plugin-neon', b'/home/pokybuild/yocto-worker/eclipse-plugin-neon/build/build', b'stable/sumo-nmut', b'git://git.yoctoproject.org/poky-contrib', b'-s', b'', b'-b', b'', b'-p', b'', b'-u', b'https://autobuilder.yoctoproject.org/typhoon/#builders/29/builds/481', b'-r', b'/srv/autobuilder/autobuilder.yoctoproject.org/pub/non-release/20190128-11/testresults', b'-q'], attempting to kill process killed by signal 9 https://autobuilder.yoctoproject.org/typhoon/#/builders/29/builds/481