Bug 8241

(Not sure where in bugzilla this belongs ... autobuilder or automated-runtime-tests?)

It seems that a qemu "ended unexpectedly" in a mips64 testimage run (https://autobuilder.yoctoproject.org/main/builders/nightly-mips64/builds/130/steps/Running%20Sanity%20Tests/logs/stdio). In the next testimage run (on the same machine) nightly-arm-lsb could not start vnc because the lock file is still there.

Bug 8210 (Should be able to restart a crashed vnc server) already exists but I'm filing this for the initial qemu (or vnc?) crash


This could be related to bug 8234 (testimage failures) or even 8061 ( Occasional qemux86 segfault).



. ./oe-init-build-env; netstat -an | grep -q 5901 ;if [ $? -ne 0 ]; then echo "Starting a VNC server on :1"; vncserver :1; else echo "Will use a VNC server already running on :1"; fi;echo 'INHERIT += "testimage"' >> ./conf/auto.conf;echo 'TEST_QEMUBOOT_TIMEOUT = "1500"' >> ./conf/auto.conf;DISPLAY=:1 bitbake core-image-minimal core-image-sato core-image-sato-sdk -c testimage
 in dir /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build (timeout 14800 secs)
 watching logfiles {}
 argv: . ./oe-init-build-env; netstat -an | grep -q 5901 ;if [ $? -ne 0 ]; then echo "Starting a VNC server on :1"; vncserver :1; else echo "Will use a VNC server already running on :1"; fi;echo 'INHERIT += "testimage"' >> ./conf/auto.conf;echo 'TEST_QEMUBOOT_TIMEOUT = "1500"' >> ./conf/auto.conf;DISPLAY=:1 bitbake core-image-minimal core-image-sato core-image-sato-sdk -c testimage
 environment:
  ADTQA_INST_PUBLISH_DIR=adt-installer-QA
  ADTREPO_DEV_PATH=/srv/www/vhosts/adtrepo-dev/
  ADTREPO_DEV_POPULATE=True
  ADTREPO_DEV_URL=http://adtrepo-dev.yoctoproject.org/
  ADTREPO_GENERATE_DEV_INSTALLER=True
  ADTREPO_GENERATE_INSTALLER=False
  ADTREPO_PATH=/srv/www/vhosts/adtrepo.yoctoproject.org/
  ADTREPO_POPULATE=False
  ADTREPO_URL=http://adtrepo.yoctoproject.org/
  ADT_INST_PUBLISH_DIR=adt-installer
  BA_PUBLISH_DIR=build-appliance
  BB_NUMBER_THREADS=16
  BUILD_HISTORY_COLLECT=True
  BUILD_HISTORY_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/buildhistory
  BUILD_HISTORY_REPO=ssh://git@git.yoctoproject.org/poky-buildhistory
  BUILD_HISTORY_WEB=True
  BUILD_HISTORY_WEB_DIR=/home/pokybuild/buildhistory-web
  BUILD_HISTORY_WHITELIST={'git://git.yoctoproject.org/poky':['master']}
  BUILD_PUBLISH_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/pub
  CLEAN_SOURCE_DIR=False
  DEB_PUBLISH_DIR=deb
  DEVKERNEL=linux-yocto
  DEVKERNEL_MUT_REPO={'git://git.yoctoproject.org/poky-contrib':['stage/master_under_test', 'sgw/mut', 'ross/mut'], 'git://git.yoctoproject.org/poky':['ross/mut']}
  DL_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/current_sources
  EMGD_DRIVER_DIR=/tmp/yocto-autobuilder/emgd-driver
  ERROR_REPORT_COLLECT=True
  GEN_IMG_MD5=True
  HOME=/home/pokybuild
  IMAGE_FSTYPES=tar.gz
  IPK_PUBLISH_DIR=ipk
  LANG=en_US.UTF-8
  LOGNAME=pokybuild
  LSB_SSTATE_DIR=/tmp/yocto-autobuilder/lsb-sstate
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
  MACHINE_PUBLISH_DIR=machines
  MAIL=/var/mail/pokybuild
  MAINTAIN_PERSISTDB=True
  MAKEFLAGS=
  MAKELEVEL=1
  MARKED_RELEASE_DIR=releases
  MFLAGS=
  OGIT_MIRROR_DIR=/home/pokybuild/git/mirror
  OGIT_TRASH_CRON_TIME=0 0 * * *
  OGIT_TRASH_DIR=/home/pokybuild/git/trash
  OGIT_TRASH_NICE_LEVEL=19
  OLDPWD=/home/pokybuild
  OPTIMIZED_GIT_CLONE=True
  PARALLEL_MAKE=16
  PATH=/home/pokybuild/yocto-autobuilder/bin:/home/pokybuild/yocto-autobuilder/yocto-autobuilder/scripts:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PERSISTDB_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/persistdb
  PUBLISH_BUILDS=True
  PUBLISH_SOURCE_MIRROR=True
  PUBLISH_SSTATE=False
  PWD=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build
  PYTHONPATH=/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/autobuilder/:/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/:/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/autobuilder/buildsteps:
  QA_MAIL_BCC=
  QA_MAIL_CC=elizabeth.flanagan@intel.com, alexandru.c.georgescu@intel.com, otavio@ossystems.com.br, yi.zhao@windriver.com
  QA_MAIL_SIG=\nThis is an automated message from\nThe Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder\nEmail: elizabeth.flanagan@intel.com
  QA_MAIL_TO=yocto@yoctoproject.org
  QEMU_PUBLISH_DIR=/machines/qemu
  RELEASE_PUBLISH_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/pub/releases
  RESOLVED_TRIGGERED_HEAD=True
  RESOLVE_TRIGGERED_HEAD=True
  RPM_PUBLISH_DIR=rpm
  SHELL=/bin/bash
  SHLVL=1
  SOURCE_DL_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/current_sources
  SOURCE_MIRROR_URL=/srv/www/vhosts/autobuilder.yoctoproject.org/pub/sources
  SOURCE_PUBLISH_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/current_sources
  SOURCE_SSTATE_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate
  SSTATE_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate
  SSTATE_PUBLISH_DIR=/srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate
  TERM=xterm
  TOASTER_UPLOAD_URL=http://toaster-ab-dev.yoctoproject.org/orm/eventfile
  USER=pokybuild
  WEB_ROOT=/srv/www/vhosts/autobuilder.yoctoproject.org/
  WEB_URL=http://autobuilder.yoctoproject.org/
  WORKERBASEDIR=/home/pokybuild/yocto-autobuilder/yocto-worker
  X8664TC_PUBLISH_DIR=toolchain/x86_64
  X86TC_PUBLISH_DIR=toolchain/i686
  XDG_SESSION_COOKIE=cce5631d6cf011d310ac1e7d532dd80e-1438592224.258724-563411987
  YOCTO_AB_CONFIG=/home/pokybuild/yocto-autobuilder/buildset-config/yoctoAB.conf
  _=./yocto-start-autobuilder
 using PTY: False

### Shell environment set up for builds. ###

You can now run 'bitbake <target>'

Common targets are:
    core-image-minimal
    core-image-sato
    meta-toolchain
    adt-installer
    meta-ide-support

You can also run generated qemu images with a command like 'runqemu qemux86'
Will use a VNC server already running on :1
Parsing recipes...done.
Parsing of 893 .bb files complete (0 cached, 893 parsed). 1319 targets, 70 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.27.1"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Debian-7.8"
TARGET_SYS        = "mips64-poky-linux"
MACHINE           = "qemumips64"
DISTRO            = "poky"
DISTRO_VERSION    = "1.8+snapshot-20150902"
TUNE_FEATURES     = "n64 bigendian fpu-hard"
TARGET_FPU        = ""
meta              
meta-yocto        
meta-yocto-bsp    = "master-next:565637ae3b2ed09c682454287d0cff1f7c6e4375"
meta-qt3          = "master:3016129d90b7ac8517a5227d819f10ad417b5b45"

NOTE: Preparing RunQueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 316 of 318 (ID: 2, /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-sato/images/core-image-sato-sdk.bb, do_testimage)
NOTE: Running task 317 of 318 (ID: 1, /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-sato/images/core-image-sato.bb, do_testimage)
NOTE: Running task 318 of 318 (ID: 0, /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-core/images/core-image-minimal.bb, do_testimage)
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Started
NOTE: recipe core-image-sato-sdk-1.0-r0: task do_testimage: Started
NOTE: recipe core-image-sato-1.0-r0: task do_testimage: Started
ERROR: Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201509012006_qemu
ERROR: Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201509012006_qemu
ERROR: Qemu log output from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20150902030545:

ERROR: Function failed: core-image-sato - FAILED to start qemu - check the task log and the boot log
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/temp/log.do_testimage.25594
NOTE: recipe core-image-sato-1.0-r0: task do_testimage: Failed
ERROR: Task 1 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-sato/images/core-image-sato.bb, do_testimage) failed with exit code '1'
ERROR: Qemu log output from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/work/qemumips64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20150902030545:
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 4.1.6-yocto-standard (pokybuild@debian7) (gcc version 4.9.3 (GCC) ) #1 PREEMPT Tue Sep 1 18:42:53 PDT 2015
earlycon: Early serial console at I/O port 0x3f8 (options '38400n8')
bootconsole [uart0] enabled
CPU0 revision is: 000182a0 (MIPS 20Kc)
FPU revision is: 000f8200
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Software DMA cache coherency enabled
Determined physical RAM map:
 memory: 0000000000001000 @ 0000000000000000 (reserved)
 memory: 00000000000ef000 @ 0000000000001000 (ROM data)
 memory: 0000000000ae9000 @ 00000000000f0000 (reserved)
 memory: 000000000f426000 @ 0000000000bd9000 (usable)
User-defined physical RAM map:
 memory: 0000000010000000 @ 0000000000000000 (usable)
Initrd not found or empty - disabling initrd
Zone ranges:
  DMA      [mem 0x0000000000000000-0x0000000000ffffff]
  Normal   [mem 0x0000000001000000-0x000000000fffffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000000000000-0x000000000fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
Primary instruction cache 32kB, VIVT, 4-way, linesize 32 bytes.
Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 64512
Kernel command line: root=/dev/hda rw console=ttyS0 console=tty ip=192.168.7.18::192.168.7.17:255.255.255.0 mem=256M rootfstype=ext4  console=tty1 console=ttyS0,115200n8
PID hash table entries: 1024 (order: 1, 8192 bytes)
Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
Enable cache parity protection for MIPS 20KC/25KF CPUs.
Memory: 245472K/262144K available (7439K kernel code, 763K rwdata, 1684K rodata, 380K init, 801K bss, 16672K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Preemptible hierarchical RCU implementation.
	Additional per-CPU info printed with stalls.
NR_IRQS:256

ERROR: Function failed: core-image-minimal - FAILED to start qemu - check the task log and the boot log
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/work/qemumips64-poky-linux/core-image-minimal/1.0-r0/temp/log.do_testimage.25595
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Failed
ERROR: Task 0 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-core/images/core-image-minimal.bb, do_testimage) failed with exit code '1'
ERROR: Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201509012006_qemu
ERROR: Function failed: core-image-sato-sdk - FAILED to start qemu - check the task log and the boot log
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato-sdk/1.0-r0/temp/log.do_testimage.25593
NOTE: recipe core-image-sato-sdk-1.0-r0: task do_testimage: Failed
ERROR: Task 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-sato/images/core-image-sato-sdk.bb, do_testimage) failed with exit code '1'
NOTE: Tasks Summary: Attempted 318 tasks of which 315 didn't need to be rerun and 3 failed.
NOTE: The errors for this build are stored in /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/log/error-report/error_report_20150902030545.txt
You can send the errors to a reports server by running:
  send-error-report /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/build/tmp/log/error-report/error_report_20150902030545.txt [-s server]
NOTE: The contents of these logs will be posted in public if you use the above command with the default server. Please ensure you remove any identifying or proprietary information when prompted before sending.

Summary: 3 tasks failed:
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-sato/images/core-image-sato.bb, do_testimage
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-core/images/core-image-minimal.bb, do_testimage
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips64/build/meta/recipes-sato/images/core-image-sato-sdk.bb, do_testimage
Summary: There were 8 ERROR messages shown, returning a non-zero exit code.
program finished with exit code 1
elapsedTime=254.760638