Bug 7055

Summary:	Fedora 21 builder can't ping qemu guests
Product:	[Infrastructure] AutoBuilder	Reporter:	Ross Burton <ross.burton>
Component:	autobuilder	Assignee:	Michael Halstead <mhalstead>
Status:	VERIFIED FIXED	QA Contact:	Alexandru Georgescu <alexandru.c.georgescu>
Severity:	major
Priority:	High	CC:	alexandru.c.georgescu, elizabeth.flanagan, infras.ab.watcher, Infras.watcher, ionutx.chisanovici, richard.purdie, sgw
Version:	unspecified
Target Milestone:	1.8 M2
Hardware:	All
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Ross Burton 2014-12-09 21:34:56 UTC

Almost every time the autobuilder tries to run sanity tests on fedora21 they fail, in particular core-image-minimal appears to fail to configure networking.

For example, https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/125/steps/Running%20Sanity%20Tests/logs/stdio

core-image-minimal fails to ping, but core-image-sato boots and the tests run (but fail).

To demonstrate that its not the image that is broken https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/126/steps/Running%20Sanity%20Tests/logs/stdio is a very similar build (mainly with the cause of the failing tests removed).

It's not easily possible to request builds to happen on specific machines so I haven't yet demonstrated the same build failing on F21 but not F20, but the pattern is clear from https://autobuilder.yoctoproject.org/main/buildslaves/fedora21.osl.yoctoproject.org?numbuilds=50 - the only builds that passed where the ones without sanity tests.

Comment 1 Michael Halstead 2014-12-10 05:25:17 UTC

taps 0 through 4 had lost their addresses.

8: tap0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 500
    link/ether da:56:00:20:aa:07 brd ff:ff:ff:ff:ff:ff
9: tap1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 500
    link/ether 52:df:81:8a:fc:b9 brd ff:ff:ff:ff:ff:ff
10: tap2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 500
    link/ether ca:38:be:c4:a4:e7 brd ff:ff:ff:ff:ff:ff
11: tap3: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 500
    link/ether a6:6f:31:d3:65:9f brd ff:ff:ff:ff:ff:ff
12: tap4: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 500
    link/ether 6a:1e:63:e6:e0:40 brd ff:ff:ff:ff:ff:ff
13: tap5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 500
    link/ether 86:29:6c:88:5b:4b brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.11/32 brd 192.168.7.255 scope global tap5
       valid_lft forever preferred_lft forever

After removing and recreating the tap devs the tests pass but the tap devices used for testing lose their addresses. 

Please share any ideas you have as to why this might be happening. I suspect NetworkManager and will try disabling it and testing again.

Comment 2 Michael Halstead 2014-12-11 21:07:13 UTC

After disabling Networkmanager the addresses remain set but the tests still fail. Now the Fedora21 is released I think I will do a fresh install and check for the problem again.

Comment 3 Ionut Chisanovici 2015-01-30 12:25:56 UTC

Hi guys,

Try this:

$sudo visudo
search the line 'Defaults    requiretty' and disable it.
At the bottom of the file (last line - it is very important to be the last line in the file otherwise will not work) add the user that runs Autobuilder service so it can run sudo commands without password:
autobuilder     ALL=(ALL)       NOPASSWD: ALL

Also, take a look over firewall rules (iptables) if enabled and selinux. We have this two services disabled on our AB environment.\

Cheers,
Ionut C

Comment 4 Ross Burton 2015-01-30 16:01:20 UTC

Michael must have figured out what the problem was a while ago as this isn't a problem any more.

Comment 5 Michael Halstead 2015-01-30 18:41:45 UTC

I'm not reopening this as it seems the installing the final release fixed the problem. I do want to note some things about the production autobuilder.

We never give the build user superuser privileges via sudo or any other way. The only privilege related thing we do for the build user is create the network tap devices owned by the build user. 

We keep SELinux enabled. The build should run with SELinux enabled on CentOS 6 and 7 and on all current versions of Fedora. If it doesn't we want to find the problem.

We do disable the host firewall. All builders sit behind a dedicated firewall.

Comment 6 Alexandru Georgescu 2015-02-25 12:13:23 UTC

Verified as per Michael's last comments.