4216 2013-04-08 07:22:11 +0000 [Autobuilder] "smart help" sanity test failed 2013-04-09 12:38:03 +0000 1 1 11 Runtime Package Management Issues package-management-issues unspecified x86 Multiple RESOLVED FIXED Medium+ normal 1.4 M6 1 bogdan.a.marinescu richard.purdie richard.purdie --- --- --- oldest_to_newest 31649 0 bogdan.a.marinescu 2013-04-08 07:22:11 +0000 http://autobuilder.yoctoproject.org:8011/builders/nightly-arm-lsb/builds/95/steps/Running%20Sanity%20Tests_1/logs/stdio | qemuarm | Test_Info: We can ssh on 192.168.7.4 within 0 seconds | spawn ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@192.168.7.4 /sbin/reboot | Warning: Permanently added '192.168.7.4' (RSA) to the list of known hosts. | root@192.168.7.4's password: | Test_Info: Shutdown Test PASS | NOTE: Test Result for qemuarm core-image-sato | NOTE: Testcase PASS FAIL NORESULT | NOTE: SSH 1 0 0 | NOTE: SCP 1 0 0 | NOTE: dmesg 1 0 0 | NOTE: smart_help 0 1 0 | NOTE: smart_query 1 0 0 | NOTE: rpm_query 1 0 0 | NOTE: connman 1 0 0 | NOTE: shutdown 1 0 0 | DEBUG: Python function do_qemuimagetest_standalone finished | ERROR: Function failed: Some testcases fail, pls. check test result and test log!!! ERROR: Task 1 (/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-arm-lsb/build/meta/recipes-sato/images/core-image-sato.bb, do_qemuimagetest_standalone) failed with exit code '1' Summary: There was 1 ERROR message shown, returning a non-zero exit code. 31666 1 richard.purdie 2013-04-08 11:52:47 +0000 We keep seeing an occasional failure in this sanity test or the following one, the smart avahi query. Sometimes these are accumpanied by: | error: db_init:db3.c:1098: dbenv->failchk(-30973): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery | Re-opening dbenv with DB_RECOVER ... | BDB2526 Finding last valid log LSN: file: 2 offset 803530 recovery 17% completeBDB1518 Recovery complete at Mon Apr 8 10:57:11 2013 | BDB1519 Maximum transaction ID 0 recovery checkpoint [2][809174] | . | recovery succeeded. | Updating cache... ######################################## [100%] | avahi-daemon-0.6.31-r6.1@mips32 | avahi-locale-en-gb-0.6.31-r6.1@mips32 | Saving cache... I was able to reproduce a segfault in "smart query avahi*" once and when this happened, looking in the dmesg logs we see: python[6080]: segfault at 6d33042c ip 4d28f6bb sp bfb106e0 error 6 in libdb-5.3.so[4d223000+133000] So it looking likely this bug is related to our other random segfault issue, in this case it just breaks the smart sanity test. 31675 2 richard.purdie 2013-04-08 13:41:29 +0000 Manged to cause a double fault on qemux86-64 which suggests this is a kernel or much more likely qemu issue: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready >udevd[475]: starting version 182 btrfs: open /dev/hdc failed btrfs: open /dev/hdc failed hrtimer: interrupt took 4848420 ns double fault: 0000 [#1] PREEMPT SMP Modules linked in: uvesafb cfbfillrect cfbimgblt cfbcopyarea CPU 0 Pid: 1143, comm: python Not tainted 3.8.4-yocto-standard #1 Bochs Bochs RIP: 0010:[<ffffffff8169e5c8>] [<ffffffff8169e5c8>] do_general_protection+0x28/0x160 RSP: 0000:ffff880005b8ff28 EFLAGS: 66862002 RAX: 00007f1a66c0bc80 RBX: 0000000000000000 RCX: 00000000004005ae RDX: 0000000009691a75 RSI: 0000000000000000 RDI: ffff880005b8ff58 RBP: ffff880005b8ff48 R08: 00007f1a671fb9f0 R09: 0000000000000001 R10: 0000000000000008 R11: 0000000000000000 R12: ffff880005b8ff58 R13: 0000000000000000 R14: 0000000009691a75 R15: 00007f1a671fb9f0 FS: 0000000000000000(0000) GS:ffff880007c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f1a668796a8 CR3: 0000000006f92000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000 Process python (pid: 1143, threadinfo ffff880005b8e000, task ffff8800001aad40) Stack: 0000000009691a75 0000000000000000 00000000004005ae 00007f1a66872cc8 00007fff09a17140 ffffffff8169e072 00007f1a671fb9f0 0000000009691a75 00007f1a66872cc8 00000000004005ae 00007fff09a17140 0000fe34cd0db6a8 Call Trace: [<ffffffff8169e072>] general_protection+0x22/0x30 Code: 00 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 20 4c 89 65 f0 49 89 fc 4c 89 6d f8 49 89 f5 48 89 5d e8 f6 87 91 00 00 00 02 74 01 <fb> 65 48 8b 1c 25 c0 b7 00 00 41 f6 84 24 88 00 00 00 03 74 3b RIP [<ffffffff8169e5c8>] do_general_protection+0x28/0x160 RSP <ffff880005b8ff28> ---[ end trace 12273eae38fc59b0 ]--- 31676 3 richard.purdie 2013-04-08 13:42:17 +0000 Also managed to get a core dump from a python smart segfault: root@qemux86-64:~# gdb python core GNU gdb (GDB) 7.5.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-poky-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/python...Reading symbols from /usr/bin/.debug/python2.7...done. done. [New LWP 786] warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `python /usr/bin/smart query avahi*'. Program terminated with signal 11, Segmentation fault. #0 0x00007fff6cd1c998 in ?? () (gdb) bt #0 0x00007fff6cd1c998 in ?? () #1 0x00007f8649cb9120 in ?? () from /usr/lib/libpython2.7.so.1.0 #2 0x00000000004006d8 in ?? () #3 0x00007fff6cd1c990 in ?? () #4 0x0000000000000004 in ?? () #5 0x00007f8649a03c8a in Py_Main (argc=4, argv=0x7fff6cd1c998) at Modules/main.c:266 #6 0x00007f8649566a35 in __libc_start_main (main=0x4006d0 <main>, argc=4, ubp_av=0x7fff6cd1c998, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff6cd1c988) at libc-start.c:258 #7 0x0000000000400701 in _start () at ../sysdeps/x86_64/start.S:123 (gdb) so looks like stack corruption occurred. 31677 4 richard.purdie 2013-04-08 13:44:32 +0000 I've been able to reproduce this failure by running: #!/bin/bash ulimit -c unlimited while [ "1" != "0" ]; do smart --help > /dev/null if [ $? -ne 0 ]; then exit 1 fi smart query avahi* if [ $? -ne 0 ]; then exit 1 fi done and at some point one of the two commands faults... 31701 5 richard.purdie 2013-04-08 15:19:57 +0000 Another backtrace, this time not corrupt: ./testit: line 14: 3017 Segmentation fault (core dumped) smart query avahi* root@qemux86-64:~# gdb python core GNU gdb (GDB) 7.5.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-poky-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/python...Reading symbols from /usr/bin/.debug/python2.7...done. done. [New LWP 3017] warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `python /usr/bin/smart query avahi*'. Program terminated with signal 11, Segmentation fault. #0 0x00007fef892cdf34 in __ieee754_exp_sse2 (x=-0.5) at ../sysdeps/ieee754/dbl-64/e_exp.c:91 91 ../sysdeps/ieee754/dbl-64/e_exp.c: No such file or directory. (gdb) bt #0 0x00007fef892cdf34 in __ieee754_exp_sse2 (x=-0.5) at ../sysdeps/ieee754/dbl-64/e_exp.c:91 #1 0x00007fef892e3632 in __GI___exp (x=<optimized out>) at ../sysdeps/ieee754/dbl-64/w_exp.c:26 #2 0x00007fef86fa673e in math_1 (arg=<optimized out>, func=0x7fef892e3620 <__GI___exp>, can_overflow=1) at /usr/src/debug/python/2.7.3-r0.3/Python-2.7.3/Modules/mathmodule.c:691 #3 0x00007fef8a073cd4 in call_function (oparg=<optimized out>, pp_stack=0x7ffffcc73d58) at Python/ceval.c:4009 #4 PyEval_EvalFrameEx (f=f@entry=0x763060, throwflag=throwflag@entry=0) at Python/ceval.c:2666 #5 0x00007fef8a074c67 in PyEval_EvalCodeEx (co=co@entry=0x7706b0, globals=globals@entry=0x739580, locals=locals@entry=0x739580, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=closure@entry=0x0) at Python/ceval.c:3253 #6 0x00007fef8a074dc2 in PyEval_EvalCode (co=co@entry=0x7706b0, globals=globals@entry=0x739580, locals=locals@entry=0x739580) at Python/ceval.c:667 #7 0x00007fef8a0849dd in PyImport_ExecCodeModuleEx (name=name@entry=0x7ffffcc760b0 "random", co=co@entry=0x7706b0, pathname=pathname@entry=0x7ffffcc73f60 "/usr/lib/python2.7/random.pyo") at Python/import.c:681 #8 0x00007fef8a084c6c in load_source_module (name=name@entry=0x7ffffcc760b0 "random", pathname=0x7ffffcc73f60 "/usr/lib/python2.7/random.pyo", pathname@entry=0x7ffffcc74fe0 "/usr/lib/python2.7/random.py", fp=<optimized out>) at Python/import.c:1018 #9 0x00007fef8a08570c in load_module (name=name@entry=0x7ffffcc760b0 "random", fp=<optimized out>, pathname=pathname@entry=0x7ffffcc74fe0 "/usr/lib/python2.7/random.py", type=<optimized out>, loader=<optimized out>) at Python/import.c:1822 #10 0x00007fef8a085c8e in import_submodule (mod=mod@entry=0x7fef8a323660 <_Py_NoneStruct>, subname=subname@entry=0x7ffffcc760b0 "random", fullname=fullname@entry=0x7ffffcc760b0 "random") at Python/import.c:2595 #11 0x00007fef8a085f00 in load_next (mod=mod@entry=0x7fef8a323660 <_Py_NoneStruct>, altmod=0x7fef8a323660 <_Py_NoneStruct>, p_name=p_name@entry=0x7ffffcc76098, buf=buf@entry=0x7ffffcc760b0 "random", p_buflen=p_buflen@entry=0x7ffffcc760a8) at Python/import.c:2415 #12 0x00007fef8a08643f in import_module_level (name=0x0, name@entry=0x766504 "random", globals=globals@entry=0x7397c0, fromlist=fromlist@entry=0x7fef8a3f2310, level=level@entry=-1, locals=<optimized out>) at Python/import.c:2136 #13 0x00007fef8a0869ea in PyImport_ImportModuleLevel (name=0x766504 "random", globals=0x7397c0, locals=<optimized out>, fromlist=0x7fef8a3f2310, level=-1) at Python/import.c:2188 #14 0x00007fef8a06c4df in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at Python/bltinmodule.c:49 #15 0x00007fef89fda0ae in PyObject_Call (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f3520, kw=<optimized out>) at Objects/abstract.c:2529 #16 0x00007fef8a06e107 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f3520, kw=kw@entry=0x0) at Python/ceval.c:3890 #17 0x00007fef8a070001 in PyEval_EvalFrameEx (f=f@entry=0x760470, throwflag=throwflag@entry=0) at Python/ceval.c:2333 #18 0x00007fef8a074c67 in PyEval_EvalCodeEx (co=co@entry=0x767c30, globals=globals@entry=0x7397c0, locals=locals@entry=0x7397c0, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=closure@entry=0x0) at Python/ceval.c:3253 #19 0x00007fef8a074dc2 in PyEval_EvalCode (co=co@entry=0x767c30, globals=globals@entry=0x7397c0, locals=locals@entry=0x7397c0) at Python/ceval.c:667 #20 0x00007fef8a0849dd in PyImport_ExecCodeModuleEx (name=name@entry=0x7ffffcc79706 "tempfile", co=co@entry=0x767c30, pathname=pathname@entry=0x7ffffcc775b0 "/usr/lib/python2.7/tempfile.pyo") at Python/import.c:681 #21 0x00007fef8a084c6c in load_source_module (name=name@entry=0x7ffffcc79706 "tempfile", pathname=0x7ffffcc775b0 "/usr/lib/python2.7/tempfile.pyo", pathname@entry=0x7ffffcc78630 "/usr/lib/python2.7/tempfile.py", fp=<optimized out>) at Python/import.c:1018 #22 0x00007fef8a08570c in load_module (name=name@entry=0x7ffffcc79706 "tempfile", fp=<optimized out>, pathname=pathname@entry=0x7ffffcc78630 "/usr/lib/python2.7/tempfile.py", type=<optimized out>, loader=<optimized out>) at Python/import.c:1822 #23 0x00007fef8a085c8e in import_submodule (mod=mod@entry=0x7fef8a323660 <_Py_NoneStruct>, subname=subname@entry=0x7ffffcc79706 "tempfile", fullname=fullname@entry=0x7ffffcc79706 "tempfile") at Python/import.c:2595 #24 0x00007fef8a085f7a in load_next (mod=mod@entry=0x7fef8a45cb40, altmod=0x7fef8a323660 <_Py_NoneStruct>, p_name=p_name@entry=0x7ffffcc796e8, buf=buf@entry=0x7ffffcc79700 "smart.tempfile", p_buflen=p_buflen@entry=0x7ffffcc796f8) at Python/import.c:2419 #25 0x00007fef8a08643f in import_module_level (name=0x0, name@entry=0x7fef8a3e0834 "tempfile", globals=globals@entry=0x758a80, fromlist=fromlist@entry=0x7fef8a323660 <_Py_NoneStruct>, level=level@entry=-1, locals=<optimized out>) at Python/import.c:2136 #26 0x00007fef8a0869ea in PyImport_ImportModuleLevel (name=0x7fef8a3e0834 "tempfile", globals=0x758a80, locals=<optimized out>, fromlist=0x7fef8a323660 <_Py_NoneStruct>, level=-1) at Python/import.c:2188 #27 0x00007fef8a06c4df in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at Python/bltinmodule.c:49 #28 0x00007fef89fda0ae in PyObject_Call (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f31b0, kw=<optimized out>) at Objects/abstract.c:2529 #29 0x00007fef8a06e107 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fef8a558dd0, arg=arg@entry=0x7fef8a3f31b0, kw=kw@entry=0x0) at Python/ceval.c:3890 #30 0x00007fef8a070001 in PyEval_EvalFrameEx (f=f@entry=0x6ef190, throwflag=throwflag@entry=0) at Python/ceval.c:2333 31704 6 richard.purdie 2013-04-08 15:23:51 +0000 *** Bug 4217 has been marked as a duplicate of this bug. *** 31707 7 richard.purdie 2013-04-08 15:35:57 +0000 Jsut to tie together all the pieces into one place, I did post an email to the mailing list about this problem: http://lists.linuxtogo.org/pipermail/openembedded-core/2013-April/037994.html and Khem replied he has seensegfaults in ld.so on qemux86-64. This may or may not be related to this bug. Other segfault like failures we've seen are: a) dmesg failure, x86-64 matchbox-panel[449]: segfault at 1 ip 00007fb0269de543 sp 00007fff0503d540 error 4 in libglib-2.0.so.0.3400.3[7fb0269a8000+121000] http://autobuilder.yoctoproject.org:8011/builders/nightly-x86-64/builds/91/steps/Running%20Sanity%20Tests/logs/stdio b) smart help failure, arm-lsb http://autobuilder.yoctoproject.org:8011/builders/nightly-arm-lsb/builds/95/steps/Running%20Sanity%20Tests_1/logs/stdio c) dmesg failure, x86-64-lsb traps: modprobe[546] general protection ip:7ff6f2b114e7 sp:7fff87cbc698 error:0 in ld-2.17.so[7ff6f2afa000+21000] http://autobuilder.yoctoproject.org:8011/builders/nightly-x86-64-lsb/builds/87/steps/Running%20Sanity%20Tests/logs/stdio d) mdesg failure, x86 rpc.mountd[458]: segfault at 0 ip (null) sp bfedb5e4 error 4 in rpc.mountd[8047000+16000] http://autobuilder.yoctoproject.org:8011/builders/nightly-x86/builds/92/steps/Running%20Sanity%20Tests/logs/stdio Running the code in comment 4 on a qemu86-64 machine locally reproduces the segfault eventually usually within an hour, often in about 10 minutes. Subsequent commands work ok, assuming the segfault didn't take out a futext and prevent access to the rpm database which did happen in one case. When its crashed for me locally I've see: a) A double fault b) "Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 474: elf_machine_rela_relative: Assertion `((reloc->r_info) & 0xffffffff) == 8' failed!" c) Various other sefgaults, some with stack corruption, some without 31761 8 richard.purdie 2013-04-09 12:38:03 +0000 With any luck this is fixed in master with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=6ec99ee2f1d90be6c115c74fb7a5bdb2969c4dba