We keep seeing an occasional failure in this sanity test or the following one, the smart avahi query. Sometimes these are accumpanied by:

| error: db_init:db3.c:1098: dbenv->failchk(-30973): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
| Re-opening dbenv with DB_RECOVER ...
| BDB2526 Finding last valid log LSN: file: 2 offset 803530
recovery 17% completeBDB1518 Recovery complete at Mon Apr  8 10:57:11 2013
| BDB1519 Maximum transaction ID 0 recovery checkpoint [2][809174]
| .
| recovery succeeded.
| Updating cache...               ######################################## [100%]
| avahi-daemon-0.6.31-r6.1@mips32
| avahi-locale-en-gb-0.6.31-r6.1@mips32
| Saving cache...

I was able to reproduce a segfault in "smart query avahi*" once and when this happened, looking in the dmesg logs we see:

python[6080]: segfault at 6d33042c ip 4d28f6bb sp bfb106e0 error 6 in libdb-5.3.so[4d223000+133000]

So it looking likely this bug is related to our other random segfault issue, in this case it just breaks the smart sanity test.

Manged to cause a double fault on qemux86-64 which suggests this is a kernel or much more likely qemu issue:

IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>udevd[475]: starting version 182
btrfs: open /dev/hdc failed
btrfs: open /dev/hdc failed
hrtimer: interrupt took 4848420 ns
double fault: 0000 [#1] PREEMPT SMP 
Modules linked in: uvesafb cfbfillrect cfbimgblt cfbcopyarea
CPU 0 
Pid: 1143, comm: python Not tainted 3.8.4-yocto-standard #1 Bochs Bochs
RIP: 0010:[<ffffffff8169e5c8>]  [<ffffffff8169e5c8>] do_general_protection+0x28/0x160
RSP: 0000:ffff880005b8ff28  EFLAGS: 66862002
RAX: 00007f1a66c0bc80 RBX: 0000000000000000 RCX: 00000000004005ae
RDX: 0000000009691a75 RSI: 0000000000000000 RDI: ffff880005b8ff58
RBP: ffff880005b8ff48 R08: 00007f1a671fb9f0 R09: 0000000000000001
R10: 0000000000000008 R11: 0000000000000000 R12: ffff880005b8ff58
R13: 0000000000000000 R14: 0000000009691a75 R15: 00007f1a671fb9f0
FS:  0000000000000000(0000) GS:ffff880007c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a668796a8 CR3: 0000000006f92000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
Process python (pid: 1143, threadinfo ffff880005b8e000, task ffff8800001aad40)
Stack:
 0000000009691a75 0000000000000000 00000000004005ae 00007f1a66872cc8
 00007fff09a17140 ffffffff8169e072 00007f1a671fb9f0 0000000009691a75
 00007f1a66872cc8 00000000004005ae 00007fff09a17140 0000fe34cd0db6a8
Call Trace:
 [<ffffffff8169e072>] general_protection+0x22/0x30
Code: 00 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 20 4c 89 65 f0 49 89 fc 4c 89 6d f8 49 89 f5 48 89 5d e8 f6 87 91 00 00 00 02 74 01 <fb> 65 48 8b 1c 25 c0 b7 00 00 41 f6 84 24 88 00 00 00 03 74 3b 
RIP  [<ffffffff8169e5c8>] do_general_protection+0x28/0x160
 RSP <ffff880005b8ff28>
---[ end trace 12273eae38fc59b0 ]---

Also managed to get a core dump from a python smart segfault:

root@qemux86-64:~# gdb python core 
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-poky-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...Reading symbols from /usr/bin/.debug/python2.7...done.
done.
[New LWP 786]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `python /usr/bin/smart query avahi*'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fff6cd1c998 in ?? ()
(gdb) bt
#0  0x00007fff6cd1c998 in ?? ()
#1  0x00007f8649cb9120 in ?? () from /usr/lib/libpython2.7.so.1.0
#2  0x00000000004006d8 in ?? ()
#3  0x00007fff6cd1c990 in ?? ()
#4  0x0000000000000004 in ?? ()
#5  0x00007f8649a03c8a in Py_Main (argc=4, argv=0x7fff6cd1c998) at Modules/main.c:266
#6  0x00007f8649566a35 in __libc_start_main (main=0x4006d0 <main>, argc=4, ubp_av=0x7fff6cd1c998, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fff6cd1c988) at libc-start.c:258
#7  0x0000000000400701 in _start () at ../sysdeps/x86_64/start.S:123
(gdb)

so looks like stack corruption occurred.

I've been able to reproduce this failure by running:

#!/bin/bash
ulimit -c unlimited
while [ "1" != "0" ]; do
    smart --help > /dev/null
    if [ $? -ne 0 ]; then
        exit 1
    fi
    smart query avahi*
    if [ $? -ne 0 ]; then
        exit 1
    fi
done

and at some point one of the two commands faults...

Another backtrace, this time not corrupt:

./testit: line 14:  3017 Segmentation fault      (core dumped) smart query avahi*
root@qemux86-64:~# gdb python core 
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-poky-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...Reading symbols from /usr/bin/.debug/python2.7...done.
done.
[New LWP 3017]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `python /usr/bin/smart query avahi*'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fef892cdf34 in __ieee754_exp_sse2 (x=-0.5) at ../sysdeps/ieee754/dbl-64/e_exp.c:91
91	../sysdeps/ieee754/dbl-64/e_exp.c: No such file or directory.
(gdb) bt
#0  0x00007fef892cdf34 in __ieee754_exp_sse2 (x=-0.5) at ../sysdeps/ieee754/dbl-64/e_exp.c:91
#1  0x00007fef892e3632 in __GI___exp (x=<optimized out>) at ../sysdeps/ieee754/dbl-64/w_exp.c:26
#2  0x00007fef86fa673e in math_1 (arg=<optimized out>, func=0x7fef892e3620 <__GI___exp>, can_overflow=1)
    at /usr/src/debug/python/2.7.3-r0.3/Python-2.7.3/Modules/mathmodule.c:691
#3  0x00007fef8a073cd4 in call_function (oparg=<optimized out>, pp_stack=0x7ffffcc73d58) at Python/ceval.c:4009
#4  PyEval_EvalFrameEx (f=f@entry=0x763060, throwflag=throwflag@entry=0) at Python/ceval.c:2666
#5  0x00007fef8a074c67 in PyEval_EvalCodeEx (co=co@entry=0x7706b0, globals=globals@entry=0x739580, locals=locals@entry=0x739580, args=args@entry=0x0, 
    argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=closure@entry=0x0)
    at Python/ceval.c:3253
#6  0x00007fef8a074dc2 in PyEval_EvalCode (co=co@entry=0x7706b0, globals=globals@entry=0x739580, locals=locals@entry=0x739580) at Python/ceval.c:667
#7  0x00007fef8a0849dd in PyImport_ExecCodeModuleEx (name=name@entry=0x7ffffcc760b0 "random", co=co@entry=0x7706b0, 
    pathname=pathname@entry=0x7ffffcc73f60 "/usr/lib/python2.7/random.pyo") at Python/import.c:681
#8  0x00007fef8a084c6c in load_source_module (name=name@entry=0x7ffffcc760b0 "random", pathname=0x7ffffcc73f60 "/usr/lib/python2.7/random.pyo", 
    pathname@entry=0x7ffffcc74fe0 "/usr/lib/python2.7/random.py", fp=<optimized out>) at Python/import.c:1018
#9  0x00007fef8a08570c in load_module (name=name@entry=0x7ffffcc760b0 "random", fp=<optimized out>, 
    pathname=pathname@entry=0x7ffffcc74fe0 "/usr/lib/python2.7/random.py", type=<optimized out>, loader=<optimized out>) at Python/import.c:1822
#10 0x00007fef8a085c8e in import_submodule (mod=mod@entry=0x7fef8a323660 <_Py_NoneStruct>, subname=subname@entry=0x7ffffcc760b0 "random", 
    fullname=fullname@entry=0x7ffffcc760b0 "random") at Python/import.c:2595
#11 0x00007fef8a085f00 in load_next (mod=mod@entry=0x7fef8a323660 <_Py_NoneStruct>, altmod=0x7fef8a323660 <_Py_NoneStruct>, p_name=p_name@entry=0x7ffffcc76098, 
    buf=buf@entry=0x7ffffcc760b0 "random", p_buflen=p_buflen@entry=0x7ffffcc760a8) at Python/import.c:2415
#12 0x00007fef8a08643f in import_module_level (name=0x0, name@entry=0x766504 "random", globals=globals@entry=0x7397c0, fromlist=fromlist@entry=0x7fef8a3f2310, 
    level=level@entry=-1, locals=<optimized out>) at Python/import.c:2136
#13 0x00007fef8a0869ea in PyImport_ImportModuleLevel (name=0x766504 "random", globals=0x7397c0, locals=<optimized out>, fromlist=0x7fef8a3f2310, level=-1)
    at Python/import.c:2188
#14 0x00007fef8a06c4df in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at Python/bltinmodule.c:49
#15 0x00007fef89fda0ae in PyObject_Call (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f3520, kw=<optimized out>) at Objects/abstract.c:2529
#16 0x00007fef8a06e107 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f3520, kw=kw@entry=0x0) at Python/ceval.c:3890
#17 0x00007fef8a070001 in PyEval_EvalFrameEx (f=f@entry=0x760470, throwflag=throwflag@entry=0) at Python/ceval.c:2333
#18 0x00007fef8a074c67 in PyEval_EvalCodeEx (co=co@entry=0x767c30, globals=globals@entry=0x7397c0, locals=locals@entry=0x7397c0, args=args@entry=0x0, 
    argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=closure@entry=0x0)
    at Python/ceval.c:3253
#19 0x00007fef8a074dc2 in PyEval_EvalCode (co=co@entry=0x767c30, globals=globals@entry=0x7397c0, locals=locals@entry=0x7397c0) at Python/ceval.c:667
#20 0x00007fef8a0849dd in PyImport_ExecCodeModuleEx (name=name@entry=0x7ffffcc79706 "tempfile", co=co@entry=0x767c30, 
    pathname=pathname@entry=0x7ffffcc775b0 "/usr/lib/python2.7/tempfile.pyo") at Python/import.c:681
#21 0x00007fef8a084c6c in load_source_module (name=name@entry=0x7ffffcc79706 "tempfile", pathname=0x7ffffcc775b0 "/usr/lib/python2.7/tempfile.pyo", 
    pathname@entry=0x7ffffcc78630 "/usr/lib/python2.7/tempfile.py", fp=<optimized out>) at Python/import.c:1018
#22 0x00007fef8a08570c in load_module (name=name@entry=0x7ffffcc79706 "tempfile", fp=<optimized out>, 
    pathname=pathname@entry=0x7ffffcc78630 "/usr/lib/python2.7/tempfile.py", type=<optimized out>, loader=<optimized out>) at Python/import.c:1822
#23 0x00007fef8a085c8e in import_submodule (mod=mod@entry=0x7fef8a323660 <_Py_NoneStruct>, subname=subname@entry=0x7ffffcc79706 "tempfile", 
    fullname=fullname@entry=0x7ffffcc79706 "tempfile") at Python/import.c:2595
#24 0x00007fef8a085f7a in load_next (mod=mod@entry=0x7fef8a45cb40, altmod=0x7fef8a323660 <_Py_NoneStruct>, p_name=p_name@entry=0x7ffffcc796e8, 
    buf=buf@entry=0x7ffffcc79700 "smart.tempfile", p_buflen=p_buflen@entry=0x7ffffcc796f8) at Python/import.c:2419
#25 0x00007fef8a08643f in import_module_level (name=0x0, name@entry=0x7fef8a3e0834 "tempfile", globals=globals@entry=0x758a80, 
    fromlist=fromlist@entry=0x7fef8a323660 <_Py_NoneStruct>, level=level@entry=-1, locals=<optimized out>) at Python/import.c:2136
#26 0x00007fef8a0869ea in PyImport_ImportModuleLevel (name=0x7fef8a3e0834 "tempfile", globals=0x758a80, locals=<optimized out>, 
    fromlist=0x7fef8a323660 <_Py_NoneStruct>, level=-1) at Python/import.c:2188
#27 0x00007fef8a06c4df in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at Python/bltinmodule.c:49
#28 0x00007fef89fda0ae in PyObject_Call (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f31b0, kw=<optimized out>) at Objects/abstract.c:2529
#29 0x00007fef8a06e107 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f31b0, kw=kw@entry=0x0) at Python/ceval.c:3890
#30 0x00007fef8a070001 in PyEval_EvalFrameEx (f=f@entry=0x6ef190, throwflag=throwflag@entry=0) at Python/ceval.c:2333

*** Bug 4217 has been marked as a duplicate of this bug. ***

Jsut to tie together all the pieces into one place, I did post an email to the mailing list about this problem:

http://lists.linuxtogo.org/pipermail/openembedded-core/2013-April/037994.html

and Khem replied he has seensegfaults in ld.so on qemux86-64. This may or may not be related to this bug. Other segfault like failures we've seen are:

a) dmesg failure, x86-64

matchbox-panel[449]: segfault at 1 ip 00007fb0269de543 sp 00007fff0503d540 error 4 in libglib-2.0.so.0.3400.3[7fb0269a8000+121000]
http://autobuilder.yoctoproject.org:8011/builders/nightly-x86-64/builds/91/steps/Running%20Sanity%20Tests/logs/stdio

b) smart help failure, arm-lsb

http://autobuilder.yoctoproject.org:8011/builders/nightly-arm-lsb/builds/95/steps/Running%20Sanity%20Tests_1/logs/stdio

c) dmesg failure, x86-64-lsb

traps: modprobe[546] general protection ip:7ff6f2b114e7 sp:7fff87cbc698 error:0 in ld-2.17.so[7ff6f2afa000+21000]
http://autobuilder.yoctoproject.org:8011/builders/nightly-x86-64-lsb/builds/87/steps/Running%20Sanity%20Tests/logs/stdio

d) mdesg failure, x86

rpc.mountd[458]: segfault at 0 ip   (null) sp bfedb5e4 error 4 in rpc.mountd[8047000+16000]
http://autobuilder.yoctoproject.org:8011/builders/nightly-x86/builds/92/steps/Running%20Sanity%20Tests/logs/stdio


Running the code in comment 4 on a qemu86-64 machine locally reproduces the segfault eventually usually within an hour, often in about 10 minutes. Subsequent commands work ok, assuming the segfault didn't take out a futext and prevent access to the rpm database which did happen in one case. When its crashed for me locally I've see:

a) A double fault
b) "Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 474: elf_machine_rela_relative: Assertion `((reloc->r_info) & 0xffffffff) == 8' failed!"
c) Various other sefgaults, some with stack corruption, some without

With any luck this is fixed in master with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6ec99ee2f1d90be6c115c74fb7a5bdb2969c4dba