Bug 11254 - runqemu tap cleanup broken when interrupted
Summary: runqemu tap cleanup broken when interrupted
Status: RESOLVED OBSOLETE
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.4 M4
Assignee: Robert Yang
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-03-27 14:30 UTC by Jussi Kukkonen
Modified: 2018-10-12 16:38 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: Regression (Used to work)
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jussi Kukkonen 2017-03-27 14:30:10 UTC
If I let runqemu create it's own tap interface and never interrupt it, it works fine.

If I interrupt runqemu with ctrl-C, then the tap interface is left there (but without ip). Further runs of runqemu will reuse the tap but will not set the ip address and in fact will remove it on shutdown even if I manually add an ip to the interface. 

Here's an example of
 * runqemu qemux86
 * ctrl-c
 * runqemu qemux86
In the end the interface does not have IP address.


jku@theory:~/src/poky/build$ runqemu qemux86
runqemu - INFO - Assuming MACHINE = qemux86
runqemu - INFO - Running MACHINE=qemux86 bitbake -e...
runqemu - INFO - MACHINE: qemux86
runqemu - INFO - DEPLOY_DIR_IMAGE: /mnt/extra-ssd/tmp/deploy/images/qemux86
runqemu - INFO - Running ls -t /mnt/extra-ssd/tmp/deploy/images/qemux86/*.qemuboot.conf...
runqemu - INFO - CONFFILE: /mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.qemuboot.conf
runqemu - INFO - Continuing with the following parameters:

KERNEL: [/mnt/extra-ssd/tmp/deploy/images/qemux86/bzImage--4.10.2+git0+69bfa87234_827a1164b1-r0-qemux86-20170327113404.bin]
MACHINE: [qemux86]
FSTYPE: [ext4]
ROOTFS: [/mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.rootfs.ext4]
CONFFILE: [/mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.qemuboot.conf]

runqemu - INFO - Running /bin/ip link...
runqemu - INFO - Found /tmp/qemu-tap-locks/tap0.skip, skipping tap0
runqemu - INFO - Found /tmp/qemu-tap-locks/tap1.skip, skipping tap1
runqemu - INFO - Found /tmp/qemu-tap-locks/tap2.skip, skipping tap2
runqemu - INFO - Found /tmp/qemu-tap-locks/tap3.skip, skipping tap3
runqemu - INFO - Setting up tap interface under sudo
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap4.lock...
runqemu - INFO - Created tap: tap4
runqemu - INFO - Network configuration: 192.168.7.10::192.168.7.9:255.255.255.0
runqemu - INFO - Running ldd /mnt/extra-ssd/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-i386...
runqemu - INFO - Running /mnt/extra-ssd/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-i386 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:0a -netdev tap,id=net0,ifname=tap4,script=no,downscript=no -drive file=/mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.rootfs.ext4,if=virtio,format=raw -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci   -cpu qemu32 -m 256 -serial mon:vc -serial null -kernel /mnt/extra-ssd/tmp/deploy/images/qemux86/bzImage--4.10.2+git0+69bfa87234_827a1164b1-r0-qemux86-20170327113404.bin -append 'root=/dev/vda rw highres=off  mem=256M ip=192.168.7.10::192.168.7.9:255.255.255.0 vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1 '
^Crunqemu - INFO - Running sudo /home/jku/src/poky/scripts/runqemu-ifdown tap4 /mnt/extra-ssd/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native
TUNSETIFF: Device or resource busy
runqemu - INFO - Releasing lockfile for tap device 'tap4'
Traceback (most recent call last):
  File "/home/jku/src/poky/scripts/runqemu", line 1199, in <module>
    ret = main()
  File "/home/jku/src/poky/scripts/runqemu", line 1192, in main
    config.start_qemu()
  File "/home/jku/src/poky/scripts/runqemu", line 1123, in start_qemu
    if subprocess.call(cmd, shell=True) != 0:
  File "/usr/lib/python3.4/subprocess.py", line 539, in call
    return p.wait(timeout=timeout)
  File "/usr/lib/python3.4/subprocess.py", line 1566, in wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib/python3.4/subprocess.py", line 1514, in _try_wait
    (pid, sts) = _eintr_retry_call(os.waitpid, self.pid, wait_flags)
  File "/usr/lib/python3.4/subprocess.py", line 491, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt
jku@theory:~/src/poky/build$ qemu-system-i386: terminating on signal 2
^C
jku@theory:~/src/poky/build$ runqemu qemux86
runqemu - INFO - Assuming MACHINE = qemux86
runqemu - INFO - Running MACHINE=qemux86 bitbake -e...
runqemu - INFO - MACHINE: qemux86
runqemu - INFO - DEPLOY_DIR_IMAGE: /mnt/extra-ssd/tmp/deploy/images/qemux86
runqemu - INFO - Running ls -t /mnt/extra-ssd/tmp/deploy/images/qemux86/*.qemuboot.conf...
runqemu - INFO - CONFFILE: /mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.qemuboot.conf
runqemu - INFO - Continuing with the following parameters:

KERNEL: [/mnt/extra-ssd/tmp/deploy/images/qemux86/bzImage--4.10.2+git0+69bfa87234_827a1164b1-r0-qemux86-20170327113404.bin]
MACHINE: [qemux86]
FSTYPE: [ext4]
ROOTFS: [/mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.rootfs.ext4]
CONFFILE: [/mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.qemuboot.conf]

runqemu - INFO - Running /bin/ip link...
runqemu - INFO - Found /tmp/qemu-tap-locks/tap0.skip, skipping tap0
runqemu - INFO - Found /tmp/qemu-tap-locks/tap1.skip, skipping tap1
runqemu - INFO - Found /tmp/qemu-tap-locks/tap2.skip, skipping tap2
runqemu - INFO - Found /tmp/qemu-tap-locks/tap3.skip, skipping tap3
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap4.lock...
runqemu - INFO - Using preconfigured tap device tap4
runqemu - INFO - If this is not intended, touch /tmp/qemu-tap-locks/tap4.skip to make runqemu skip tap4.
runqemu - INFO - Network configuration: 192.168.7.10::192.168.7.9:255.255.255.0
runqemu - INFO - Running ldd /mnt/extra-ssd/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-i386...
runqemu - INFO - Running /mnt/extra-ssd/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-i386 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:0a -netdev tap,id=net0,ifname=tap4,script=no,downscript=no -drive file=/mnt/extra-ssd/tmp/deploy/images/qemux86/core-image-minimal-qemux86-20170327134906.rootfs.ext4,if=virtio,format=raw -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci   -cpu qemu32 -m 256 -serial mon:vc -serial null -kernel /mnt/extra-ssd/tmp/deploy/images/qemux86/bzImage--4.10.2+git0+69bfa87234_827a1164b1-r0-qemux86-20170327113404.bin -append 'root=/dev/vda rw highres=off  mem=256M ip=192.168.7.10::192.168.7.9:255.255.255.0 vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1 '
Comment 1 Jussi Kukkonen 2017-03-27 14:51:41 UTC
This is especially nasty because the oe-selftest sends SIGTERM to runqemu and that has the same result: in a test like runtime-test.Postinst.test_postinst_rootfs_and_boot only the first of six qemu tests succeeds because on the five subsequent ones there's no ip on the tap interface.
Comment 2 Jussi Kukkonen 2017-03-28 08:01:38 UTC
I don't know which component to blame so I'm documenting the oe-selftest things here as well:

qemurunner.py says this after :
> Sending SIGTERM to runqemu

Here runqemu does not run BaseConfig.cleanup() as one would expect, so "tunctl -d tapN" is never executed.



Has this ever worked? Am I doing something wrong?
Comment 3 Jussi Kukkonen 2017-03-31 09:03:06 UTC
> TUNSETIFF: Device or resource busy

This is the issue, I think. "tunctl -d tapN" returns EBUSY (or at least the ioctl does). 

I'll experiment with retrying tunctl but the real solution would probably be that qemurunner/runqemu copes with this situation...
Comment 4 Jussi Kukkonen 2017-03-31 12:51:12 UTC
Retrying "tunctl -d" on failure does make this work when using runqemu from shell.

It does nothing to fix the qemurunner case because, as far as I can tell, runqemu is killed before it gets to do "tunctl -d" -- and something still removes the address from the interface.

In case it was not clear the same problem happens when using pregenerated interfaces: when runqemu is killed, something removes the address from the interface.
Comment 5 Robert Yang 2017-04-06 03:29:57 UTC
The tag creating is very fast, I tried several times to interrupt it during creating tap, but failed, Jussi Kukkonen, do you have any updates, please ?
Comment 6 Jussi Kukkonen 2017-04-06 07:42:40 UTC
There is nothing that needs quick reactions in my case:
 * "runqemu <machine>", let the image boot if you want
 * ctrl-c runqemu

After this there are two bugs as I understand it:
 1. the tap interface still exists even though it was created by runqemu (it should be removed but cleanup never run in this case)
 2. the tap interface loses the ip address very soon after ctrl-c

As a result next runqemu invocation will use an interface that already exists but has no IP address.

Bug #1 is probably there for everyone and has always been: it just hasn't been very exposed until #2... Bug #2 is something I think didn't use to happen for me -- I have no idea if it's a yocto issue or maybe something specific to build host setup (Debian Unstable).
Comment 7 Robert Yang 2017-07-13 03:07:34 UTC
Hi Jussi,

Would you please try again ? The Ctrl-C doesn't  break Qemu in my testing.
Comment 8 Jussi Kukkonen 2017-07-13 06:37:14 UTC
(In reply to comment #7)
> Hi Jussi,
> 
> Would you please try again ? The Ctrl-C doesn't  break Qemu in my testing.

It might not happen 100% every time, but yeah, still reproduces.

Are you saying that after 'ctrl-C' in runqemu, the tap interface created by runqemu gets properly removed for you?
Comment 9 Robert Yang 2017-09-04 09:35:42 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > Hi Jussi,
> > 
> > Would you please try again ? The Ctrl-C doesn't  break Qemu in my testing.
> 
> It might not happen 100% every time, but yeah, still reproduces.
> 
> Are you saying that after 'ctrl-C' in runqemu, the tap interface created by
> runqemu gets properly removed for you?

Yes, it did. I will close the bug, and please reopen it if you can reproduce it again.