1489 2011-09-16 11:22:57 +0000 tcf-agent segmentaion fault cause the re-running remote debug fail in Eclipse 2012-07-25 13:47:08 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core unspecified x86 Multiple VERIFIED FIXED (P3)need to test the latest tcf 1.0 upstream whether the problem still exists Medium normal 1.3 M2 1 jessica.zhang lianhao.lu alexandru.c.georgescu hongnax.xu meta.mr.watcher meta.watcher song.liu yp.ep.watcher yp.watcher --- --- --- oldest_to_newest 16270 0 jessica.zhang 2011-09-16 11:22:57 +0000 Create a Yocto C project, run remote debugging once successfully, switch back to C perspective, try to remote debug again, you'll run into: "Launching YHWC_gdb_i586-poky-linux" has encountered a problem, Could not create the hostShellProcess 16272 1 jessica.zhang 2011-09-16 14:16:17 +0000 I got the following exceptions: org.eclipse.rse.services.clientserver.messages.SystemMessageException: Command [C 79 Terminals launch "vt100" null []] aborted at org.eclipse.tm.internal.tcf.rse.TCFRSETask.getS(TCFRSETask.java:48) at org.eclipse.tm.internal.tcf.rse.shells.TCFTerminalShell.<init>(TCFTerminalShell.java:240) at org.eclipse.tm.internal.tcf.rse.terminals.TCFTerminalService.launchTerminal(TCFTerminalService.java:38) at org.eclipse.rse.internal.services.shells.TerminalShellService.runCommand(TerminalShellService.java:65) at org.eclipse.rse.internal.services.shells.TerminalShellService.launchShell(TerminalShellService.java:53) at org.eclipse.rse.services.shells.AbstractShellService.launchShell(AbstractShellService.java:46) at org.eclipse.cdt.launch.remote.RSEHelper.remoteShellExec(RSEHelper.java:266) at org.eclipse.cdt.launch.remote.launching.RemoteRunLaunchDelegate.launch(RemoteRunLaunchDelegate.java:129) at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:854) at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:703) at org.eclipse.debug.internal.ui.DebugUIPlugin.buildAndLaunch(DebugUIPlugin.java:928) at org.eclipse.debug.internal.ui.DebugUIPlugin$8.run(DebugUIPlugin.java:1132) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) 16284 2 lianhao.lu 2011-09-19 04:43:45 +0000 It seems the tcf-agent in the target encountered a segmentation fault in malloc(). Bisecting the upstream svn found that revision r1287 seems ok, but the revision r1290 has this problem. Currently the upstream evision r1855 which is used by oe-core seems having the same problem as r1290. Needs to dig more into what's been changed between r1287 and r1290. 16305 3 lianhao.lu 2011-09-20 00:14:33 +0000 Looked at the difference between r1287 and r1290. It seems the difference was that it improved the event queue performance by reducing the number pthread_mutex lock/unlock calls. The patch itself seemed fine to me and I'm guessing this multi-thread concurrency improvement exposed some other memory overwrite bugs. Also, if we increase the pthread stack size from 32KB (which is set in the tcf-agent) to 8MB(linux default), it seems hard to reproduce this bug. Gdb the r1290 tcf-agent in the qemux86 target, the stack frames were something like the following when segmentation fault happened: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb7c06b70 (LWP 8350)] 0xb7ce6e43 in __GI___libc_malloc (bytes=20) at malloc.c:3615 3615 { (gdb) bt #0 0xb7ce6e43 in __GI___libc_malloc (bytes=20) at malloc.c:3615 #1 0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622 #2 0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622 #3 0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622 ... ... (hundreds of the same frame info) ... ... #786 0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622 #787 0x0804e8a1 in loc_alloc_zero (size=20) at framework/myalloc.c:49 #788 0x0805436b in post_event_with_delay (handler=0x80521e7 <tcp_channel_read_done>, arg=0x80cf218, delay=0) at framework/events.c:105 #789 0x0805469c in post_event (handler=0x80521e7 <tcp_channel_read_done>, arg=0x80cf218) at framework/events.c:155 #790 0x0804ffe6 in worker_thread_handler (x=0x80d2368) at framework/asyncreq.c:184 #791 0xb7fbe997 in start_thread (arg=0xb7c06b70) at pthread_create.c:304 #792 0xb7d4227e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:133 16309 4 lianhao.lu 2011-09-20 05:32:19 +0000 Running the tcf-agent on emenlow or on my linux desktop, can NOT reproduce this bug. 16329 5 song.liu 2011-09-20 11:22:30 +0000 According to Mark, this problem was found on ARM also, and it's fixed for ARM. We can verify this and see if it's already been fixed on other archs. If we cannot reproduce, please mark it as resolved for QA to verify this. 16334 6 jessica.zhang 2011-09-20 15:05:31 +0000 I pulled down the latest 1.1 RC branch and rebuilt core-image-sato-sdk, seems the problem is gone with the newly built image. Checked with Mark on IRC, seems the problem is triggered by the prelink issue which is fixed in the RC branch. 16345 7 lianhao.lu 2011-09-20 20:39:05 +0000 This seems not prelink related. I tried the qemux86 core-image-sato-sdk image without prelink, the problem still there. I also compile the tcf-agent in the target qemux86 and met the same problem. Anyhow, on qemux86-64 the problem seems not reproducible. 16351 8 lianhao.lu 2011-09-21 01:32:55 +0000 Built the qemux86 core-image-sato-sdk image on branch 1.1M4, confirmed that this bug is still there. 16361 9 jessica.zhang 2011-09-21 13:05:42 +0000 Lianhao, what's your host to build the image, mine's 32bit. I tried today, my agent still able to handle multiple remote debug without seg faulting... 16372 10 lianhao.lu 2011-09-21 18:09:39 +0000 My building machine is a 32bit FC13, test machine is 64bit Ubuntu 10.04. 16409 11 jessica.zhang 2011-09-22 14:18:39 +0000 when tried to create the core file for tcf-agent, I did ulimit -c unlimited, then ulimit -a, then I start tcf-agent (all of this in core-image-sato-sdk for qemux86), not my remote debug and work reliably again, which I haven't able to capture the core file yet, can you try on your end? 16415 12 jessica.zhang 2011-09-22 14:34:10 +0000 my trace gave me program terminated with signal 11, Segmentation fault #0 0x4559b011 in malloc_atfork (sz=100, caller=0x4559afe5) at arena.c:198 and read this link http://bytes.com/topic/c/answers/215865-disadvantages-malloc 16567 13 lianhao.lu 2011-09-29 19:09:38 +0000 Created an upstream bug entry at https://bugs.eclipse.org/bugs/show_bug.cgi?id=359502 19317 14 lianhao.lu 2012-03-22 01:53:27 +0000 wait upstream https://bugs.eclipse.org/bugs/show_bug.cgi?id=359502 22967 15 jessica.zhang 2012-07-10 17:44:10 +0000 Lianhao and Hongna, Can you verify if this issue still exist, if not please close the bug 23190 16 hongnax.xu 2012-07-18 03:03:45 +0000 jessica, i don't meet this issue in 1.3_M2 RC1 testing, git branch: poky/master git info: eb0cb7e8234f5d2e5623406e9660be91cf52f65e, but i can't close it cause there is no close states in my options. 23226 17 jessica.zhang 2012-07-18 17:15:11 +0000 can't reproduce the bug any more 23484 18 alexandru.c.georgescu 2012-07-25 13:46:40 +0000 I tried today with Juno and the latest eclipse-poky. Couldn't reproduce. 23485 19 alexandru.c.georgescu 2012-07-25 13:47:08 +0000 setting the bug to verified-fixed