8234 – testimage failure on autobuilder

Bug 8234 - testimage failure on autobuilder

Summary: testimage failure on autobuilder

Status:	VERIFIED FIXED

Alias:	None

Product:	Runtime Testing
Classification:	QA/Testing
Component:	general (show other bugs)
Version:	unspecified
Hardware:	x86 Multiple

Importance:	High normal
Target Milestone:	2.0
Assignee:	Richard Purdie
QA Contact:	Bogdan Alexandru Voiculescu

URL:
Whiteboard:

Depends on:
Blocks:	8249
	Show dependency tree

Reported:	2015-09-01 14:03 UTC by Jussi Kukkonen
Modified:	2015-10-23 13:48 UTC (History)
CC List:	3 users (show)

See Also:
OS type for building Yocto:	---
Type of Regression:	---
Verified:
Documentation change:	No (bug/feature does not impact docs)

Attachments
Sanity Tests stdio from nightly-arm-lsb (117.28 KB, application/octet-stream) 2015-09-01 14:03 UTC, Jussi Kukkonen	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Jussi Kukkonen 2015-09-01 14:03:20 UTC

Created attachment 2699 [details]
Sanity Tests stdio from nightly-arm-lsb

Weird testimage failure for nightly-arm-lsb core-image-lsb-sdk where it seems to start running the tests just fine but fails almost every test. Lots of scp failures with "connect to host 192.168.7.20 port 22: No route to host". 

core-image-lsb tests succeed, only core-image-lsb-sdk ones failed.

The nightly in question was killed by RP but this build had finished before that: https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/460


Attaching the full log, excerpt below


Build Configuration:
BB_VERSION        = "1.27.1"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Debian-7.8"
TARGET_SYS        = "arm-poky-linux-gnueabi"
MACHINE           = "qemuarm"
DISTRO            = "poky-lsb"
DISTRO_VERSION    = "1.8+snapshot-20150901"
TUNE_FEATURES     = "arm armv5 thumb dsp"
TARGET_FPU        = "soft"
meta              
meta-yocto        
meta-yocto-bsp    = "master:22afc047dd1b8f83e56c4a9862710776c509e3c5"
meta-qt3          = "master:3016129d90b7ac8517a5227d819f10ad417b5b45"

NOTE: Preparing RunQueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 316 of 317 (ID: 1, /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/recipes-extended/images/core-image-lsb-sdk.bb, do_testimage)
NOTE: Running task 317 of 317 (ID: 0, /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/recipes-extended/images/core-image-lsb.bb, do_testimage)
NOTE: recipe core-image-lsb-sdk-1.0-r0: task do_testimage: Started
NOTE: recipe core-image-lsb-1.0-r0: task do_testimage: Started
ERROR: Function failed: core-image-lsb-sdk - FAILED - check the task log and the ssh log
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-lsb-sdk/1.0-r0/temp/log.do_testimage.31710

Comment 1 Jussi Kukkonen 2015-09-01 17:00:16 UTC

Same thing is now happening with nightly-arm: core-image-sato testimage succeeds but core-image-sato-sdk testimage fails.


BTW: I filed this on automated-runtime-testing but it may be something in the  image itself that fails -- I just didn't know what to file it on.

Comment 2 Jussi Kukkonen 2015-09-02 07:02:07 UTC

So far this seems to have appeared at least on arm, arm-lsb and mips-lsb: the test for the actual image always succeeds but -sdk fails.

Sometimes the image boots fine and starts the tests, then stops responding (as in the attached log). Sometimes there's no output at all (https://autobuilder.yoctoproject.org/main/builders/nightly-mips-lsb/builds/468/steps/Running%20Sanity%20Tests/logs/stdio):

> command timed out: 14800 seconds without output, attempting to kill

I'm assuming these are the same issue.



(by the way, 14800 secs of no output! maybe we should try to fail earlier)

Comment 3 Richard Purdie 2015-09-03 15:28:57 UTC

qemu seems to be segfaulting. If qemu exits, the tests should really stop, not continue like this.

We're in the process of enabling core dumps on the AB, then we hmay have a better handle on why this is failing.

So two fixes needed, handle qemu dieing better and the underlying problem.

Comment 4 Richard Purdie 2015-09-04 10:50:38 UTC

When the tests fail in the middle of a run, qemu has segfaulted. We enabled core dumps and the backtrace from this is:


PASTEBIN  |  #1 paste tool since 2002	

    create new pasteshop new!toolsapiarchivefaq 

PASTEBIN
create new paste     trending pastes

    sign uploginmy alertsmy settingsmy profile	

Want more features on Pastebin? Sign Up, it's FREE!
Public Pastes

    Untitled0 sec ago
    Untitled5 sec ago
    Untitled19 sec ago
    Untitled11 sec ago
    Hair Serum For Bla...33 sec ago
    UntitledBatch | 32 sec ago
    Untitled43 sec ago
    Untitled38 sec ago

0
0
Guest	
Untitled
By: a guest on Sep 4th, 2015  |  syntax: None  |  size: 5.19 KB  |  views: 0  |  expires: in 7 days
download  |  raw  |  embed  |  report abuse  |  print  |  QR code  |  clone

pokybuild@debian8:~$ gdb --core=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-lsb/1.0-r0/core-image-lsb-1.0/core /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm...done.
[New LWP 4536]
[New LWP 4534]
[New LWP 4530]
[New LWP 4537]
[New LWP 6396]
 
warning: Corrupted shared library list: 0x7f8d5f27e540 != 0x6198225000007f8d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  smc91c111_pop_tx_fifo_done (s=0x7f8d6158b560)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:179
179             s->tx_fifo_done[i] = s->tx_fifo_done[i + 1];
(gdb) bt
#0  smc91c111_pop_tx_fifo_done (s=0x7f8d6158b560)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:179
#1  smc91c111_writeb (opaque=0x7f8d6158b560, offset=12, value=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:431
#2  0x00007f8d5ecacd65 in memory_region_oldmmio_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>,
    size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:434
#3  0x00007f8d5ecac5dd in access_with_adjusted_size (addr=140245200319840, addr@entry=12, value=0xc, value@entry=0x7f8d52ac63e8,
    size=1, access_size_min=2031671516, access_size_max=32, access=0x7f8d5ecacd30 <memory_region_oldmmio_write_accessor>,
    mr=0x7f8d6158f8f0, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#4  0x00007f8d5ecae08b in memory_region_dispatch_write (mr=mr@entry=0x7f8d6158f8f0, addr=12, data=2, size=size@entry=1,
    attrs=attrs@entry=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#5  0x00007f8d5ec7b78f in address_space_rw (as=0x7f8d5f408600 <address_space_memory>, addr=268501004, attrs=...,
    buf=buf@entry=0x7f8d52ac64b0 "\002", len=1, is_write=is_write@entry=true)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2451
#6  0x00007f8d5ec7b9e0 in address_space_write (len=<optimized out>, buf=0x7f8d52ac64b0 "\002", attrs=..., addr=<optimized out>,
    as=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2521
#7  subpage_write (opaque=<optimized out>, addr=<optimized out>, value=<optimized out>, len=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2081
#8  0x00007f8d5ecac5dd in access_with_adjusted_size (addr=140245200319840, addr@entry=12, value=0xc, value@entry=0x7f8d52ac6558,
    size=1, access_size_min=2031671516, access_size_max=32, access=0x7f8d5ecac500 <memory_region_write_with_attrs_accessor>,
    mr=0x7f8d618d5750, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#9  0x00007f8d5ecae08b in memory_region_dispatch_write (mr=0x7f8d618d5750, addr=12, data=2, size=1, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#10 0x00007f8d5584b512 in ?? ()
#11 0x5245564952444f45 in ?? ()
#12 0x000000000000003d in ?? ()
#13 0x0000000000313178 in ?? ()
#14 0xfffffffe7ffbfcff in ?? ()

    #15 0x0000000000000000 in ?? ()

clone this paste RAW Paste Data
pokybuild@debian8:~$ gdb --core=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-lsb/1.0-r0/core-image-lsb-1.0/core /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm...done.
[New LWP 4536]
[New LWP 4534]
[New LWP 4530]
[New LWP 4537]
[New LWP 6396]

warning: Corrupted shared library list: 0x7f8d5f27e540 != 0x6198225000007f8d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  smc91c111_pop_tx_fifo_done (s=0x7f8d6158b560)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:179
179	        s->tx_fifo_done[i] = s->tx_fifo_done[i + 1];
(gdb) bt
#0  smc91c111_pop_tx_fifo_done (s=0x7f8d6158b560)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:179
#1  smc91c111_writeb (opaque=0x7f8d6158b560, offset=12, value=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:431
#2  0x00007f8d5ecacd65 in memory_region_oldmmio_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, 
    size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:434
#3  0x00007f8d5ecac5dd in access_with_adjusted_size (addr=140245200319840, addr@entry=12, value=0xc, value@entry=0x7f8d52ac63e8, 
    size=1, access_size_min=2031671516, access_size_max=32, access=0x7f8d5ecacd30 <memory_region_oldmmio_write_accessor>, 
    mr=0x7f8d6158f8f0, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#4  0x00007f8d5ecae08b in memory_region_dispatch_write (mr=mr@entry=0x7f8d6158f8f0, addr=12, data=2, size=size@entry=1, 
    attrs=attrs@entry=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#5  0x00007f8d5ec7b78f in address_space_rw (as=0x7f8d5f408600 <address_space_memory>, addr=268501004, attrs=..., 
    buf=buf@entry=0x7f8d52ac64b0 "\002", len=1, is_write=is_write@entry=true)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2451
#6  0x00007f8d5ec7b9e0 in address_space_write (len=<optimized out>, buf=0x7f8d52ac64b0 "\002", attrs=..., addr=<optimized out>, 
    as=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2521
#7  subpage_write (opaque=<optimized out>, addr=<optimized out>, value=<optimized out>, len=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2081
#8  0x00007f8d5ecac5dd in access_with_adjusted_size (addr=140245200319840, addr@entry=12, value=0xc, value@entry=0x7f8d52ac6558, 
    size=1, access_size_min=2031671516, access_size_max=32, access=0x7f8d5ecac500 <memory_region_write_with_attrs_accessor>, 
    mr=0x7f8d618d5750, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#9  0x00007f8d5ecae08b in memory_region_dispatch_write (mr=0x7f8d618d5750, addr=12, data=2, size=1, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#10 0x00007f8d5584b512 in ?? ()
#11 0x5245564952444f45 in ?? ()
#12 0x000000000000003d in ?? ()
#13 0x0000000000313178 in ?? ()
#14 0xfffffffe7ffbfcff in ?? ()
#15 0x0000000000000000 in ?? ()
Pastebin.com Tools & Applications
iPhone/iPad Windows Firefox Chrome WebOS Android Mac Opera Click.to UNIX WinPhone
create new paste  |  api  |  trends  |  syntax languages  |  faq  |  tools  |  privacy  |  cookies  |  contact  |  dmca  |  advertise on pastebin  |  shop NEW!  |  go pro
Follow us: pastebin on facebook  |  pastebin on twitter  |  pastebin in the news
Dedicated Server Hosting by Steadfast
Pastebin v3.11 rendered in: 0.010 seconds
Site design & logo © 2015 Pastebin; user contributions (pastes) licensed under cc by-sa 3.0	
 
pokybuild@debian8:~$ gdb --core=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-lsb/1.0-r0/core-image-lsb-1.0/core /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm...done.
[New LWP 4536]
[New LWP 4534]
[New LWP 4530]
[New LWP 4537]
[New LWP 6396]

warning: Corrupted shared library list: 0x7f8d5f27e540 != 0x6198225000007f8d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  smc91c111_pop_tx_fifo_done (s=0x7f8d6158b560)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:179
179	        s->tx_fifo_done[i] = s->tx_fifo_done[i + 1];
(gdb) bt
#0  smc91c111_pop_tx_fifo_done (s=0x7f8d6158b560)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:179
#1  smc91c111_writeb (opaque=0x7f8d6158b560, offset=12, value=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:431
#2  0x00007f8d5ecacd65 in memory_region_oldmmio_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, 
    size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:434
#3  0x00007f8d5ecac5dd in access_with_adjusted_size (addr=140245200319840, addr@entry=12, value=0xc, value@entry=0x7f8d52ac63e8, 
    size=1, access_size_min=2031671516, access_size_max=32, access=0x7f8d5ecacd30 <memory_region_oldmmio_write_accessor>, 
    mr=0x7f8d6158f8f0, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#4  0x00007f8d5ecae08b in memory_region_dispatch_write (mr=mr@entry=0x7f8d6158f8f0, addr=12, data=2, size=size@entry=1, 
    attrs=attrs@entry=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#5  0x00007f8d5ec7b78f in address_space_rw (as=0x7f8d5f408600 <address_space_memory>, addr=268501004, attrs=..., 
    buf=buf@entry=0x7f8d52ac64b0 "\002", len=1, is_write=is_write@entry=true)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2451
#6  0x00007f8d5ec7b9e0 in address_space_write (len=<optimized out>, buf=0x7f8d52ac64b0 "\002", attrs=..., addr=<optimized out>, 
    as=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2521
#7  subpage_write (opaque=<optimized out>, addr=<optimized out>, value=<optimized out>, len=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2081
#8  0x00007f8d5ecac5dd in access_with_adjusted_size (addr=140245200319840, addr@entry=12, value=0xc, value@entry=0x7f8d52ac6558, 
    size=1, access_size_min=2031671516, access_size_max=32, access=0x7f8d5ecac500 <memory_region_write_with_attrs_accessor>, 
    mr=0x7f8d618d5750, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#9  0x00007f8d5ecae08b in memory_region_dispatch_write (mr=0x7f8d618d5750, addr=12, data=2, size=1, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#10 0x00007f8d5584b512 in ?? ()
#11 0x5245564952444f45 in ?? ()
#12 0x000000000000003d in ?? ()
#13 0x0000000000313178 in ?? ()
#14 0xfffffffe7ffbfcff in ?? ()
#15 0x0000000000000000 in ?? ()

(gdb) print s->tx_fifo_done
$1 = {99614720, 99614720, 99614720, 99614720}
(gdb) print s->tx_fifo_done_len
$2 = 99614719

so it looks like tx_fifo_done_len has been corrupted.

I've posted this onto qemu-devel to see if anyone can spot anything. It also generally seems to happen on debian8. 

https://autobuilder.yoctoproject.org/main/builders/nightly-oecore/builds/477/steps/Running%20Sanity%20Tests/logs/stdio

https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/465/steps/Running%20Sanity%20Tests/logs/stdio

both gave the same coredump backtrace.

Ideally when qemu segfaults, the tests should stop, not merrily continue.

Comment 5 Richard Purdie 2015-09-04 11:31:47 UTC

Update from qemu-devel:

> That would suggest the rx_fifo buffer is overrunning (assuming
> none of the other fields in the struct look like they've
> been corrupted). Can you try putting
>     assert(s->rx_fifo_len < NUM_PACKETS);
>  before
>     s->rx_fifo[s->rx_fifo_len++] = packetnum;
> in smc91c111_receive(), and see if you hit that assertion?

(gdb) print s->tx_fifo_len
$2 = 0
(gdb) print s->rx_fifo_len
$3 = 10

So just based on that, yes, seems that the rx_fifo looks to be
overrunning. I can add the asserts but I think it would just confirm
this.

Comment 6 Jussi Kukkonen 2015-09-04 12:03:00 UTC

Filed bug 8257 "if qemu segfaults, tests should stop running".

Comment 7 Richard Purdie 2015-09-04 12:45:59 UTC

Adding the assert requested (and some others just to be sure) shows things start to go wrong in smc91c111_receive(). The backtrace from the assert:

Reading symbols from /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-arm...done.
[New LWP 18905]
[New LWP 18903]
[New LWP 18904]
[New LWP 18980]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f098fcda107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007f098fcda107 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f098fcdb4e8 in __GI_abort () at abort.c:89
#2  0x00007f098fcd3226 in __assert_fail_base (fmt=0x7f098fe09ce8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x7f0992d90e27 "s->rx_fifo_len < 4", 
    file=file@entry=0x7f0992d90ce8 "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c", line=line@entry=681, 
    function=function@entry=0x7f0992d910e0 <__PRETTY_FUNCTION__.25357> "smc91c111_receive") at assert.c:92
#3  0x00007f098fcd32d2 in __GI___assert_fail (assertion=assertion@entry=0x7f0992d90e27 "s->rx_fifo_len < 4", 
    file=file@entry=0x7f0992d90ce8 "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c", line=line@entry=681, 
    function=function@entry=0x7f0992d910e0 <__PRETTY_FUNCTION__.25357> "smc91c111_receive") at assert.c:101
#4  0x00007f0992bff36b in smc91c111_receive (nc=<optimized out>, buf=0x7f09955b7ce8 "RT", size=1514)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:681
#5  0x00007f0992c658fb in qemu_deliver_packet (sender=0x49d7, sender@entry=0x7f0995093220, flags=18905, flags@entry=0, 
    data=0x6 <error: Cannot access memory at address 0x6>, data@entry=0x7f09955b7ce8 "RT", size=18446744073709551615, 
    size@entry=1514, opaque=0xfefefefefefefeff)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/net.c:577
#6  0x00007f0992c672ad in qemu_net_queue_deliver (size=1514, data=0x7f09955b7ce8 "RT", flags=0, sender=0x7f0995093220, 
    queue=0x7f09953d8ae0)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/queue.c:157
#7  qemu_net_queue_send (queue=0x7f09953d8ae0, sender=sender@entry=0x7f0995093220, flags=flags@entry=0, 
    data=data@entry=0x7f09955b7ce8 "RT", size=size@entry=1514, sent_cb=sent_cb@entry=0x0)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/queue.c:192
#8  0x00007f0992c64e0a in qemu_send_packet_async_with_flags (sender=sender@entry=0x7f0995093220, flags=flags@entry=0, 
    buf=buf@entry=0x7f09955b7ce8 "RT", size=size@entry=1514, sent_cb=sent_cb@entry=0x0)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/net.c:640
#9  0x00007f0992c6594f in qemu_send_packet_async (sent_cb=0x0, size=size@entry=1514, buf=buf@entry=0x7f09955b7ce8 "RT", 
    sender=sender@entry=0x7f0995093220)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/net.c:652
#10 qemu_send_packet (nc=nc@entry=0x7f0995093220, buf=buf@entry=0x7f09955b7ce8 "RT", size=size@entry=1514)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/net.c:653
#11 0x00007f0992c6776b in net_hub_receive (hub=<optimized out>, len=1514, buf=0x7f09955b7ce8 "RT", source_port=0x7f0995093570)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/hub.c:55
#12 net_hub_port_receive (nc=0x7f0995093570, buf=0x7f09955b7ce8 "RT", len=1514)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/hub.c:114
#13 0x00007f0992c658fb in qemu_deliver_packet (sender=0x49d7, flags=18905, data=0x6 <error: Cannot access memory at address 0x6>, 
    data@entry=0x7f09955b7ce8 "RT", size=18446744073709551615, opaque=0xfefefefefefefeff)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/net.c:577
#14 0x00007f0992c6719a in qemu_net_queue_deliver (size=<optimized out>, data=<optimized out>, flags=<optimized out>, 
    sender=<optimized out>, queue=0x7f09950934e0)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.---Type <return> to continue, or q <return> to quit---
4.0/net/queue.c:157
#15 qemu_net_queue_flush (queue=0x7f09950934e0)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/queue.c:254
#16 0x00007f0992c67d1e in net_hub_flush (nc=0x7f0995093220)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/hub.c:350
#17 0x00007f0992c6500d in qemu_flush_or_purge_queued_packets (nc=0x7f09953d8980, purge=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/net/net.c:602
#18 0x00007f0992bffb7f in smc91c111_writeb (opaque=0x7f09952eb560, offset=<optimized out>, value=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:382
#19 0x00007f0992bffe24 in smc91c111_writew (opaque=0x7f09952eb560, offset=0, value=128)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/hw/net/smc91c111.c:612
#20 0x00007f0992a7ed65 in memory_region_oldmmio_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, 
    size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:434
#21 0x00007f0992a7e5dd in access_with_adjusted_size (addr=18903, addr@entry=0, value=0x49d9, value@entry=0x7f09868983e8, size=2, 
    access_size_min=4294967295, access_size_max=4278124287, access=0x7f0992a7ed30 <memory_region_oldmmio_write_accessor>, 
    mr=0x7f09952ef8f0, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#22 0x00007f0992a8008b in memory_region_dispatch_write (mr=mr@entry=0x7f09952ef8f0, addr=0, data=128, size=size@entry=2, 
    attrs=attrs@entry=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171
#23 0x00007f0992a4d861 in address_space_rw (as=0x7f09931da600 <address_space_memory>, addr=268500992, attrs=..., 
    buf=buf@entry=0x7f09868984b0 "\200", len=2, is_write=is_write@entry=true)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2445
#24 0x00007f0992a4d9e0 in address_space_write (len=<optimized out>, buf=0x7f09868984b0 "\200", attrs=..., addr=<optimized out>, 
    as=<optimized out>)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2521
#25 subpage_write (opaque=<optimized out>, addr=<optimized out>, value=<optimized out>, len=<optimized out>, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/exec.c:2081
#26 0x00007f0992a7e5dd in access_with_adjusted_size (addr=18903, addr@entry=0, value=0x49d9, value@entry=0x7f0986898558, size=2, 
    access_size_min=4294967295, access_size_max=4278124287, access=0x7f0992a7e500 <memory_region_write_with_attrs_accessor>, 
    mr=0x7f0995635750, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:506
#27 0x00007f0992a8008b in memory_region_dispatch_write (mr=0x7f0995635750, addr=0, data=128, size=2, attrs=...)
    at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/x86_64-linux/qemu-native/2.4.0-r1/qemu-2.4.0/memory.c:1171

Comment 8 Richard Purdie 2015-09-04 17:11:49 UTC

As far as I can tell, adding:

    if (s->rx_fifo_len >= NUM_PACKETS)
        return -1;

into the _receive function would see to fix the issue...

Comment 9 Richard Purdie 2015-09-15 09:15:59 UTC

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=fa2f7f996457cb1c506be5584178db250a3b1a2c

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ea562de57590c966cd5a75fda8defecd397e6436

(Has been discussed on the qemu-devel mailing list)

Comment 10 Bogdan Alexandru Voiculescu 2015-10-23 13:48:24 UTC

Verified with bug 8257 and 8229
No errors on build https://autobuilder.yoctoproject.org/main/builders/nightly-arm-lsb/builds/500/steps/Running%20Sanity%20Tests/logs/stdio

master:5233565c80db869f6f539f37f26ea6662d9dc5f6