Bug 8129

Summary: ip absent from qemu args on core-image-sato sanity tests (several builders)
Product: [QA/Testing] Build Testing Reporter: Alex Franco <alejandro.franco>
Component: generalAssignee: Alejandro Hernandez <alejandro.hernandez>
Status: RESOLVED FIXED QA Contact: Mihail Stanciu <stanciux.mihail>
Severity: normal    
Priority: Undecided CC: alejandro.hernandez, alexandru.c.georgescu, richard.purdie, stanciux.mihail
Version: 1.9   
Target Milestone: ---   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Bug Depends on:    
Bug Blocks: 8176    
Attachments:
Description Flags
Results for failure running sanity tests on nightly-x32 build #422
none
Results of failure running sanity tests for builder nightly-qa-skeleton build #424 none

Description Alex Franco 2015-08-07 22:41:28 UTC
Created attachment 2640 [details]
Results for failure running sanity tests on nightly-x32 build #422

qemu arguments do not include "ip=" when running Sanity Tests for:

Builder: nightly-qa-skeleton
Branch: master-next
last commit where this occurred: 005a5b60f00a898b701de6df7095adf7440acebf
See first attachment for [https://autobuilder.yoctoproject.org/main/builders/nightly-x32/builds/422/steps/Running%20Sanity%20Tests/logs/stdio]

Builder: nightly-x32
Branch: joshuagl/fido-next
last commit where this occurred: 7706ac25f38e33fe1dcebdcff7e141293308cffb

A recent patch (see the output of nightly-qa-skeleton  logs) improved
Comment 1 Alex Franco 2015-08-07 22:44:49 UTC
Created attachment 2641 [details]
Results of failure running sanity tests for builder nightly-qa-skeleton build #424

Added attachment for [https://autobuilder.yoctoproject.org/main/builders/nightly-qa-skeleton/builds/424/steps/Running%20Sanity%20Tests/logs/stdio]
Comment 2 Alejandro Hernandez 2015-08-07 23:21:40 UTC
(In reply to comment #0)
> Created attachment 2640 [details]
> Results for failure running sanity tests on nightly-x32 build #422
> 
> qemu arguments do not include "ip=" when running Sanity Tests for:
> 
> Builder: nightly-qa-skeleton
> Branch: master-next
> last commit where this occurred: 005a5b60f00a898b701de6df7095adf7440acebf
> See first attachment for
> [https://autobuilder.yoctoproject.org/main/builders/nightly-x32/builds/422/
> steps/Running%20Sanity%20Tests/logs/stdio]
> 
> Builder: nightly-x32
> Branch: joshuagl/fido-next
> last commit where this occurred: 7706ac25f38e33fe1dcebdcff7e141293308cffb
> 
> A recent patch (see the output of nightly-qa-skeleton  logs) improved

Ok, so this isn't happening anymore on master-next thanks to the patch submitted, we catch the exception correctly, but we still don't know why the IP string wasn't there int he first place
Comment 3 Alejandro Hernandez 2015-08-07 23:35:06 UTC
Following up, it doesn't make much sense, since the IP is passed via the runqemu script directly:

KERNEL_NETWORK_CMD="ip=192.168.7.$n2::192.168.7.$n1:255.255.255.0"

this variable is then included in $KERNCMDLINE, by checking the target's filesystem, which could potentially be causing the issue, but the cases seem to be correctly defined, in specific, these logs are from a MACHINE="qemux86-64" and a rootfstype=ext4, (which is what is checked to include $KERNEL_NETWORK_CMD in $KERNCMDLINE)
Comment 4 Alejandro Hernandez 2015-08-07 23:36:53 UTC
There must be something wrong when passing arguments to qemu since the cmdline:

/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-skeleton/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-x86_64-kernel/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-skeleton/build/build/tmp/deploy/images/qemux86-64/bzImage-qemux86-64.bin-cpucore2duo-vgavmware-no-reboot-serialtcp:127.0.0.1:57673-m256--appendvga=0 uvesafb.mode_option=640x480-32 root= rw mem=256M  oprofile.timer=1 rootfstype=ext4  console=tty1 console=ttyS0,115200n8

seems incomplete and its missing more than just the IPs
Comment 5 Richard Purdie 2015-08-09 01:22:19 UTC
It failed with Alejandro's patch applied: https://autobuilder.yoctoproject.org/main/builers/nightly-arm/builds/428/steps/Running Sanity Tests/logs/stdio

which confirms it doesn't make it onto the commandline. The question is then why? I added some more debug onto the autobuilder:

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 02ac89c..06bd389 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -100,6 +100,7 @@ class QemuRunner:
         # oe-selftest (this makes stty error out immediately). There ought
         # to be a proper fix but this will suffice for now.
         self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp)
+        output = self.runqemu.stdout
 
         logger.info("runqemu started, pid is %s" % self.runqemu.pid)
         logger.info("waiting at most %s seconds for qemu pid" % self.runqemutime)
@@ -128,7 +129,10 @@ class QemuRunner:
                     self.ip = ips[0]
                     self.server_ip = ips[1]
             except IndexError, ValueError:
-                logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used: %s" % cmdline)
+                import fcntl
+                fl = fcntl.fcntl(output, fcntl.F_GETFL)
+                fcntl.fcntl(output, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+                logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used: %s\n %s" % (cmdline, os.read(output.fileno(), 1000000)))
                 self.stop()
                 return False
             logger.info("Target IP: %s" % self.ip)

which then shows this for the same failure:

ERROR: Function failed: core-image-minimal - FAILED to start qemu - check the task log and the boot log
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/temp/log.do_testimage.6449
Log data follows:
| DEBUG: Executing python function do_testimage
| NOTE: rootfs file: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4
| NOTE: Qemu log file: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20150809011726
| NOTE: SSH log file: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/ssh_target_log.20150809011726
| NOTE: Created listening socket for qemu serial console on: 127.0.0.1:54588
| NOTE: runqemu started, pid is 6450
| NOTE: waiting at most 60 seconds for qemu pid
| NOTE: qemu started - qemu procces pid is 6509
| NOTE: Couldn't get ip from qemu process arguments! Here is the qemu command line used: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm-kernel/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/deploy/images/qemuarm/zImage-qemuarm.bin-Mversatilepb-drivefile=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4,format=raw-no-reboot-no-reboot-serialtcp:127.0.0.1:54588-m128--appendroot=/dev/sda rw console=ttyAMA0,115200 console=tty  mem=128M highres=off rootfstype=ext4  console=tty1 console=ttyS0,115200n8
|  Please use simplified serial or kvm options instead
| 
| Continuing with the following parameters:
| KERNEL: [/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/deploy/images/qemuarm/zImage-qemuarm.bin]
| ROOTFS: [/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4]
| FSTYPE: [ext4]
| stty: standard input: Inappropriate ioctl for device
| Acquiring lockfile for tap0@NONE...
| Acquiring lockfile for tap1@NONE...
| Acquiring lockfile for tap2@NONE...
| Acquiring lockfile for tap3@NONE...
| Acquiring lockfile for tap4@NONE...
| Using preconfigured tap device 'tap4@NONE'
| If this is not intended, touch /tmp/qemu-tap-locks/tap4@NONE.skip to make runqemu skip tap4@NONE.
| /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/scripts/runqemu-internal: line 256: 4@NONE: value too great for base (error token is "4@NONE")
| Running qemu-system-arm...
| /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm -kernel /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/deploy/images/qemuarm/zImage-qemuarm.bin -M versatilepb -drive file=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4,format=raw -no-reboot -no-reboot -serial tcp:127.0.0.1:54588 -m 128 --append "root=/dev/sda rw console=ttyAMA0,115200 console=tty mem=128M highres=off rootfstype=ext4 console=tty1 console=ttyS0,115200n8"
| 
| NOTE: Sending SIGTERM to runqemu
| DEBUG: Python function do_testimage finished
| ERROR: Function failed: core-image-minimal - FAILED to start qemu - check the task log and the boot log
ERROR: Task 0 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-core/images/core-image-minimal.bb, do_testimage) failed with exit code '1'
Comment 6 Richard Purdie 2015-08-09 23:14:43 UTC
Improvement to runqemu script error handling: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=8020a9e44fffb687d4372c59239958776302a8ae

Ensure log output gets seen when runqemu fails:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=37e78463e58c374ba2ff933acfc4441cbb666c1f

Actual fix for issue:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0968e3a68b0b06caeb827938ddf556f21bfce646

(turns out this only happens on fedora22)
Comment 7 Richard Purdie 2015-08-09 23:17:37 UTC
*** Bug 8112 has been marked as a duplicate of this bug. ***