Bug 12890

Summary:	[stable/rocko-next] BitbakeSelftest fails
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Armin Kuster <akuster>
Component:	core	Assignee:	Randy MacLeod <randy.macleod>
Status:	RESOLVED WORKSFORME	QA Contact:
Severity:	normal
Priority:	Medium	CC:	meta.mr.watcher, meta.watcher, randy.macleod, richard.purdie
Version:	2.4.5
Target Milestone:	2.6
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	Regression (Used to work)
Verified:		Documentation change:	Don't know

Description Armin Kuster 2018-08-15 14:18:25 UTC

======================================================================
FAIL: test_wget_checkstatus (bb.tests.fetch.FetchCheckStatusTest) (url='http://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/build/bitbake/lib/bb/tests/fetch.py", line 856, in test_wget_checkstatus
    self.assertTrue(ret, msg="URI %s, can't check status" % (u))
AssertionError: False is not true : URI http://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz, can't check status

======================================================================
FAIL: test_wget_checkstatus (bb.tests.fetch.FetchCheckStatusTest) (url='https://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/build/bitbake/lib/bb/tests/fetch.py", line 856, in test_wget_checkstatus
    self.assertTrue(ret, msg="URI %s, can't check status" % (u))
AssertionError: False is not true : URI https://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz, can't check status

======================================================================
FAIL: test_wget_checkstatus (bb.tests.fetch.FetchCheckStatusTest) (url='https://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/build/bitbake/lib/bb/tests/fetch.py", line 856, in test_wget_checkstatus
    self.assertTrue(ret, msg="URI %s, can't check status" % (u))
AssertionError: False is not true : URI https://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz, can't check status

======================================================================
FAIL: test_wget_checkstatus_connection_cache (bb.tests.fetch.FetchCheckStatusTest) (url='http://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/build/bitbake/lib/bb/tests/fetch.py", line 871, in test_wget_checkstatus_connection_cache
    self.assertTrue(ret, msg="URI %s, can't check status" % (u))
AssertionError: False is not true : URI http://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz, can't check status

======================================================================
FAIL: test_wget_checkstatus_connection_cache (bb.tests.fetch.FetchCheckStatusTest) (url='https://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/build/bitbake/lib/bb/tests/fetch.py", line 871, in test_wget_checkstatus_connection_cache
    self.assertTrue(ret, msg="URI %s, can't check status" % (u))
AssertionError: False is not true : URI https://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz, can't check status

======================================================================
FAIL: test_wget_checkstatus_connection_cache (bb.tests.fetch.FetchCheckStatusTest) (url='https://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly/build/bitbake/lib/bb/tests/fetch.py", line 871, in test_wget_checkstatus_connection_cache
    self.assertTrue(ret, msg="URI %s, can't check status" % (u))
AssertionError: False is not true : URI https://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz, can't check status

Comment 1 Armin Kuster 2018-08-15 14:18:46 UTC

This passed on my local system

Comment 2 Armin Kuster 2018-08-15 14:19:53 UTC

failure was on  tumbleweed.yocto.io

Comment 3 Richard Purdie 2018-08-15 15:58:16 UTC

I suspect this is some kind of rate limiting/DoS protection upstream as the same failures occurred in master too.

Comment 4 Armin Kuster 2018-08-16 01:48:42 UTC

so how should this be resolved?

Comment 5 Randy MacLeod 2018-08-16 15:36:17 UTC

Email to webmasters at gnu.org sent:

Subject: Intermittent failures for ftp.gnu.org connections

I work on the Yocto Project which builds custom GNU/Linux
distros from source. We have a series of build system tests
that fail intermittently for connections to:
   http[s]://ftp.gnu.org
as described, somewhat cryptically, here:
   https://bugzilla.yoctoproject.org/show_bug.cgi?id=12890

Is there some kind of rate limiting/DoS protection that
is run on the ftp.gnu.org server and if so can we get added
to a whitelist? Do you have any log file evidence that servers from the yoctoproject.org/.io are being throttled?

While we could use a local mirror for our testing, the intent
is to test the system as users of the Yocto Project would use
it. A local mirror would hide any connectivity or reliability
problems.

Comment 6 Randy MacLeod 2018-08-16 15:37:26 UTC

Armin,
Do you recall when these errors happened?
Are they worse at a particular time of day?

Comment 7 Armin Kuster 2018-08-16 15:41:44 UTC

(In reply to comment #6)
> Armin,
> Do you recall when these errors happened?
> Are they worse at a particular time of day?

Weekends or in the Evening ( PST )

Comment 8 Randy MacLeod 2018-08-16 17:21:24 UTC

A GNU webmaster replied:

   We had intermittent downtime on that machine in the last few days 
   due to problems in our datacenter, I think they could have caused 
   the errors. We also have some DDOS and request limits at the firewall 
   level, but I don't think they have changed recently. I'm closing this 
   by now, if your build system continues to see errors please reopen by 
   replying to this email, and we will look into the network/firewall 
   side of things.

Armin, 
Do you think that explains the problem or has this been happening for a while already? Is it worth trying the test again a few times?

Comment 9 Randy MacLeod 2018-09-24 18:00:07 UTC

The problem apparently was not on the webserver end so there's not much to be done. Re-open if the issue persists.