| Summary: | ip absent from qemu args on core-image-sato sanity tests (several builders) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Build Testing | Reporter: | Alex Franco <alejandro.franco> | ||||||
| Component: | general | Assignee: | Alejandro Hernandez <alejandro.hernandez> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | Mihail Stanciu <stanciux.mihail> | ||||||
| Severity: | normal | ||||||||
| Priority: | Undecided | CC: | alejandro.hernandez, alexandru.c.georgescu, richard.purdie, stanciux.mihail | ||||||
| Version: | 1.9 | ||||||||
| Target Milestone: | --- | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | |||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | Don't know | |||||||
| Bug Depends on: | |||||||||
| Bug Blocks: | 8176 | ||||||||
| Attachments: |
|
||||||||
|
Description
Alex Franco
2015-08-07 22:41:28 UTC
Created attachment 2641 [details] Results of failure running sanity tests for builder nightly-qa-skeleton build #424 Added attachment for [https://autobuilder.yoctoproject.org/main/builders/nightly-qa-skeleton/builds/424/steps/Running%20Sanity%20Tests/logs/stdio] (In reply to comment #0) > Created attachment 2640 [details] > Results for failure running sanity tests on nightly-x32 build #422 > > qemu arguments do not include "ip=" when running Sanity Tests for: > > Builder: nightly-qa-skeleton > Branch: master-next > last commit where this occurred: 005a5b60f00a898b701de6df7095adf7440acebf > See first attachment for > [https://autobuilder.yoctoproject.org/main/builders/nightly-x32/builds/422/ > steps/Running%20Sanity%20Tests/logs/stdio] > > Builder: nightly-x32 > Branch: joshuagl/fido-next > last commit where this occurred: 7706ac25f38e33fe1dcebdcff7e141293308cffb > > A recent patch (see the output of nightly-qa-skeleton logs) improved Ok, so this isn't happening anymore on master-next thanks to the patch submitted, we catch the exception correctly, but we still don't know why the IP string wasn't there int he first place Following up, it doesn't make much sense, since the IP is passed via the runqemu script directly: KERNEL_NETWORK_CMD="ip=192.168.7.$n2::192.168.7.$n1:255.255.255.0" this variable is then included in $KERNCMDLINE, by checking the target's filesystem, which could potentially be causing the issue, but the cases seem to be correctly defined, in specific, these logs are from a MACHINE="qemux86-64" and a rootfstype=ext4, (which is what is checked to include $KERNEL_NETWORK_CMD in $KERNCMDLINE) There must be something wrong when passing arguments to qemu since the cmdline: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-skeleton/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-x86_64-kernel/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-skeleton/build/build/tmp/deploy/images/qemux86-64/bzImage-qemux86-64.bin-cpucore2duo-vgavmware-no-reboot-serialtcp:127.0.0.1:57673-m256--appendvga=0 uvesafb.mode_option=640x480-32 root= rw mem=256M oprofile.timer=1 rootfstype=ext4 console=tty1 console=ttyS0,115200n8 seems incomplete and its missing more than just the IPs It failed with Alejandro's patch applied: https://autobuilder.yoctoproject.org/main/builers/nightly-arm/builds/428/steps/Running Sanity Tests/logs/stdio which confirms it doesn't make it onto the commandline. The question is then why? I added some more debug onto the autobuilder: diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py index 02ac89c..06bd389 100644 --- a/meta/lib/oeqa/utils/qemurunner.py +++ b/meta/lib/oeqa/utils/qemurunner.py @@ -100,6 +100,7 @@ class QemuRunner: # oe-selftest (this makes stty error out immediately). There ought # to be a proper fix but this will suffice for now. self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp) + output = self.runqemu.stdout logger.info("runqemu started, pid is %s" % self.runqemu.pid) logger.info("waiting at most %s seconds for qemu pid" % self.runqemutime) @@ -128,7 +129,10 @@ class QemuRunner: self.ip = ips[0] self.server_ip = ips[1] except IndexError, ValueError: - logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used: %s" % cmdline) + import fcntl + fl = fcntl.fcntl(output, fcntl.F_GETFL) + fcntl.fcntl(output, fcntl.F_SETFL, fl | os.O_NONBLOCK) + logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used: %s\n %s" % (cmdline, os.read(output.fileno(), 1000000))) self.stop() return False logger.info("Target IP: %s" % self.ip) which then shows this for the same failure: ERROR: Function failed: core-image-minimal - FAILED to start qemu - check the task log and the boot log ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/temp/log.do_testimage.6449 Log data follows: | DEBUG: Executing python function do_testimage | NOTE: rootfs file: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4 | NOTE: Qemu log file: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20150809011726 | NOTE: SSH log file: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/ssh_target_log.20150809011726 | NOTE: Created listening socket for qemu serial console on: 127.0.0.1:54588 | NOTE: runqemu started, pid is 6450 | NOTE: waiting at most 60 seconds for qemu pid | NOTE: qemu started - qemu procces pid is 6509 | NOTE: Couldn't get ip from qemu process arguments! Here is the qemu command line used: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm-kernel/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/deploy/images/qemuarm/zImage-qemuarm.bin-Mversatilepb-drivefile=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4,format=raw-no-reboot-no-reboot-serialtcp:127.0.0.1:54588-m128--appendroot=/dev/sda rw console=ttyAMA0,115200 console=tty mem=128M highres=off rootfstype=ext4 console=tty1 console=ttyS0,115200n8 | Please use simplified serial or kvm options instead | | Continuing with the following parameters: | KERNEL: [/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/deploy/images/qemuarm/zImage-qemuarm.bin] | ROOTFS: [/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4] | FSTYPE: [ext4] | stty: standard input: Inappropriate ioctl for device | Acquiring lockfile for tap0@NONE... | Acquiring lockfile for tap1@NONE... | Acquiring lockfile for tap2@NONE... | Acquiring lockfile for tap3@NONE... | Acquiring lockfile for tap4@NONE... | Using preconfigured tap device 'tap4@NONE' | If this is not intended, touch /tmp/qemu-tap-locks/tap4@NONE.skip to make runqemu skip tap4@NONE. | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/scripts/runqemu-internal: line 256: 4@NONE: value too great for base (error token is "4@NONE") | Running qemu-system-arm... | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm -kernel /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/deploy/images/qemuarm/zImage-qemuarm.bin -M versatilepb -drive file=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/core-image-minimal-qemuarm-testimage.ext4,format=raw -no-reboot -no-reboot -serial tcp:127.0.0.1:54588 -m 128 --append "root=/dev/sda rw console=ttyAMA0,115200 console=tty mem=128M highres=off rootfstype=ext4 console=tty1 console=ttyS0,115200n8" | | NOTE: Sending SIGTERM to runqemu | DEBUG: Python function do_testimage finished | ERROR: Function failed: core-image-minimal - FAILED to start qemu - check the task log and the boot log ERROR: Task 0 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-core/images/core-image-minimal.bb, do_testimage) failed with exit code '1' Improvement to runqemu script error handling: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=8020a9e44fffb687d4372c59239958776302a8ae Ensure log output gets seen when runqemu fails: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=37e78463e58c374ba2ff933acfc4441cbb666c1f Actual fix for issue: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0968e3a68b0b06caeb827938ddf556f21bfce646 (turns out this only happens on fedora22) |