Bug 8587

Is this on the QA autobuilder machines?  I wonder how this works on the main autobuilder.

(In reply to comment #1)
> Is this on the QA autobuilder machines?  I wonder how this works on the main
> autobuilder.

Yes, it's on the QA machines. 

It is my understanding that the main autobuilder doesn't have this issue, so I'm inclined to believe this is something to do with our local configuration, but I haven't been able to identify where the difference between the main AB and our AB is.

Can we have the logs (the runqemu output specifically) for the first time runqemu was executed and apparently deleted the tap device?

(In reply to comment #3)
> Can we have the logs (the runqemu output specifically) for the first time
> runqemu was executed and apparently deleted the tap device?

Continuing with the following parameters:
KERNEL: [/home/autobuilder/yocto-autobuilder/yocto-worker/nightly-mips/build/build/tmp/deploy/images/qemumips/vmlinux-qemumips.bin]
ROOTFS: [/home/autobuilder/yocto-autobuilder/yocto-worker/nightly-mips/build/build/tmp/deploy/images/qemumips/core-image-minimal-qemumips-20151029181503.rootfs.ext4]
FSTYPE: [ext4]
Acquiring lockfile for tap0...
Using preconfigured tap device 'tap0'
If this is not intended, touch /tmp/qemu-tap-locks/tap0.skip to make runqemu skip tap0.
Running qemu-system-mips...
/home/autobuilder/yocto-autobuilder/yocto-worker/nightly-mips/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-mips -kernel /home/autobuilder/yocto-autobuilder/yocto-worker/nightly-mips/build/build/tmp/deploy/images/qemumips/vmlinux-qemumips.bin -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=no,downscript=no -M malta -drive file=/home/autobuilder/yocto-autobuilder/yocto-worker/nightly-mips/build/build/tmp/deploy/images/qemumips/core-image-minimal-qemumips-20151029181503.rootfs.ext4,format=raw -no-reboot -vga cirrus -show-cursor -usb -usbdevice wacom-tablet -no-reboot -m 256 -serial mon:vc -serial null --append "root=/dev/hda rw console=ttyS0 console=tty ip=192.168.7.2::192.168.7.1:255.255.255.0 mem=256M rootfstype=ext4 "
Releasing lockfile of preconfigured tap device 'tap0'

Created attachment 2834 [details]
full runqemu output

Attaching full runqemu output with debug output on Fedora 22.

To note: interface is taken down AFTER runqemu script ends execution.

Created attachment 2835 [details]
full runqemu output ubuntu

Full runqemu debug output on Ubuntu 15.04

To note: tap interface still remains up after scripts finish running.

Investigation started.

At this point this issue seems to be caused by bad management of Fedora 22 TAP interfaces.

If NetworkManager is stopped, the issue does not reproduce. Looking into a proper fix.

NetworkManager version on Fedora 22 machine:
[autobuilder@FedoraAB ~]$ /usr/sbin/NetworkManager -V
1.0.6-6.fc22

Reproduced on Fedora 21:

[daniel@fedora-ws poky-build-M3]$ sudo NetworkManager --version
0.9.10.2-5.fc21
[daniel@fedora-ws poky-build-M3]$ cat /etc/*release*
Fedora release 21 (Twenty One)
cat: /etc/lsb-release.d: Is a directory
NAME=Fedora
VERSION="21 (Twenty One)"
ID=fedora
VERSION_ID=21
PRETTY_NAME="Fedora 21 (Twenty One)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:21"
HOME_URL="https://fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=21
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=21
Fedora release 21 (Twenty One)
Fedora release 21 (Twenty One)
cpe:/o:fedoraproject:fedora:21
[daniel@fedora-ws poky-build-M3]$ uname -a
Linux fedora-ws 4.1.8-100.fc21.x86_64 #1 SMP Tue Sep 22 12:13:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[daniel@fedora-ws poky-build-M3]$

Autobuilder wise, we've documented these issues:

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=b1f2f0cd4b29edbdb167c95238520c72d6d11649

and

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=218d57081f1ef74308c145866d3a522207d3f375

It has been suggested that the autobuilder should be setting up the tap devices. I'm very opposed to the autobuilder messing around with end users networking as there is too much that could go wrong there. The possible fixes here are to either document it (which we've done) or fix NetworkManager.

Alternative solutions:

http://support.qacafe.com/knowledge-base/how-do-i-prevent-network-manager-from-controlling-an-interface/

1. Tell Network Manager to stop controlling the interfaces used by the test configuration
2. Stop the Network Manager process
3. Remove the Network Manager software from the system

Solution 1. might be handy to implement.

*** Bug 8804 has been marked as a duplicate of this bug. ***

Pushing to M3 due to change of resources

I suspect the easiest solution is to modify uilt/path.c in qemu such that it doesn't interate all paths at the start but does so "on demand". This would avoid it walking pieces of the tree it never uses.

Comment on the wrong bug, sorry.

For NetworkManager.conf, the best solution would be to add a line in the conf 
file that marks the tap* devices as unmanaged

[keyfile]
unmanaged-devices=interface-name:tap*

(In reply to comment #18)
> For NetworkManager.conf, the best solution would be to add a line in the
> conf 
> file that marks the tap* devices as unmanaged
> 
> [keyfile]
> unmanaged-devices=interface-name:tap*

I'll look into doing this as soon as the machines in question are free. Right now they're running builds for M3.rc2.

(In reply to comment #19)
> (In reply to comment #18)
> > For NetworkManager.conf, the best solution would be to add a line in the
> > conf 
> > file that marks the tap* devices as unmanaged
> > 
> > [keyfile]
> > unmanaged-devices=interface-name:tap*
> 
> I'll look into doing this as soon as the machines in question are free.
> Right now they're running builds for M3.rc2.

The workaround seems to have done the trick. So far the tap interfaces have not been deleted by running one test.
Will leave a full set of builds to run over the weekend, but I don't expect things to fail.
Will update should there be a problem.

Confirmed everything works.

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=60786b8d90062beaa55c941075496b723eb2b4d7