Bug 1489 - tcf-agent segmentaion fault cause the re-running remote debug fail in Eclipse
Summary: tcf-agent segmentaion fault cause the re-running remote debug fail in Eclipse
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.3 M2
Assignee: Lianhao Lu
QA Contact:
URL:
Whiteboard: (P3)need to test the latest tcf 1.0 u...
Depends on:
Blocks:
 
Reported: 2011-09-16 11:22 UTC by Jessica
Modified: 2012-07-25 13:47 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jessica 2011-09-16 11:22:57 UTC
Create a Yocto C project, run remote debugging once successfully, switch back to C perspective, try to remote debug again, you'll run into:

"Launching YHWC_gdb_i586-poky-linux" has encountered a problem, Could not create the hostShellProcess
Comment 1 Jessica 2011-09-16 14:16:17 UTC
I got the following exceptions:

org.eclipse.rse.services.clientserver.messages.SystemMessageException: Command [C 79 Terminals launch "vt100" null []] aborted
	at org.eclipse.tm.internal.tcf.rse.TCFRSETask.getS(TCFRSETask.java:48)
	at org.eclipse.tm.internal.tcf.rse.shells.TCFTerminalShell.<init>(TCFTerminalShell.java:240)
	at org.eclipse.tm.internal.tcf.rse.terminals.TCFTerminalService.launchTerminal(TCFTerminalService.java:38)
	at org.eclipse.rse.internal.services.shells.TerminalShellService.runCommand(TerminalShellService.java:65)
	at org.eclipse.rse.internal.services.shells.TerminalShellService.launchShell(TerminalShellService.java:53)
	at org.eclipse.rse.services.shells.AbstractShellService.launchShell(AbstractShellService.java:46)
	at org.eclipse.cdt.launch.remote.RSEHelper.remoteShellExec(RSEHelper.java:266)
	at org.eclipse.cdt.launch.remote.launching.RemoteRunLaunchDelegate.launch(RemoteRunLaunchDelegate.java:129)
	at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:854)
	at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:703)
	at org.eclipse.debug.internal.ui.DebugUIPlugin.buildAndLaunch(DebugUIPlugin.java:928)
	at org.eclipse.debug.internal.ui.DebugUIPlugin$8.run(DebugUIPlugin.java:1132)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Comment 2 Lianhao Lu 2011-09-19 04:43:45 UTC
It seems the tcf-agent in the target encountered a segmentation fault in malloc(). Bisecting the upstream svn found that revision r1287 seems ok, but the revision r1290 has this problem. Currently the upstream evision r1855 which is used by oe-core seems having the same problem as r1290.

Needs to dig more into what's been changed between r1287 and r1290.
Comment 3 Lianhao Lu 2011-09-20 00:14:33 UTC
Looked at the difference between r1287 and r1290. It seems the difference was that it improved the event queue performance by reducing the number pthread_mutex lock/unlock calls. The patch itself seemed fine to me and I'm guessing this multi-thread concurrency improvement exposed some other memory overwrite bugs. Also, if we increase the pthread stack size from 32KB (which is set in the tcf-agent) to 8MB(linux default), it seems hard to reproduce this bug.

Gdb the r1290 tcf-agent in the qemux86 target, the stack frames were something like the following when segmentation fault happened:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7c06b70 (LWP 8350)]
0xb7ce6e43 in __GI___libc_malloc (bytes=20) at malloc.c:3615
3615	{

(gdb) bt
#0  0xb7ce6e43 in __GI___libc_malloc (bytes=20) at malloc.c:3615
#1  0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622
#2  0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622
#3  0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622
... ...
(hundreds of the same frame info)
... ...
#786 0xb7ce6fe5 in __GI___libc_malloc (bytes=20) at malloc.c:3622
#787 0x0804e8a1 in loc_alloc_zero (size=20) at framework/myalloc.c:49
#788 0x0805436b in post_event_with_delay (handler=0x80521e7 <tcp_channel_read_done>, 
    arg=0x80cf218, delay=0) at framework/events.c:105
#789 0x0805469c in post_event (handler=0x80521e7 <tcp_channel_read_done>, arg=0x80cf218)
    at framework/events.c:155
#790 0x0804ffe6 in worker_thread_handler (x=0x80d2368) at framework/asyncreq.c:184
#791 0xb7fbe997 in start_thread (arg=0xb7c06b70) at pthread_create.c:304
#792 0xb7d4227e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:133
Comment 4 Lianhao Lu 2011-09-20 05:32:19 UTC
Running the tcf-agent on emenlow or on my linux desktop, can NOT reproduce this bug.
Comment 5 Song Liu 2011-09-20 11:22:30 UTC
According to Mark, this problem was found on ARM also, and it's fixed for ARM. We can verify this and see if it's already been fixed on other archs. If we cannot reproduce, please mark it as resolved for QA to verify this.
Comment 6 Jessica 2011-09-20 15:05:31 UTC
I pulled down the latest 1.1 RC branch and rebuilt core-image-sato-sdk, seems the problem is gone with the newly built image.  Checked with Mark on IRC, seems the problem is triggered by the prelink issue which is fixed in the RC branch.
Comment 7 Lianhao Lu 2011-09-20 20:39:05 UTC
This seems not prelink related. I tried the qemux86 core-image-sato-sdk image without prelink, the problem still there. I also compile the tcf-agent in the target qemux86 and met the same problem.

Anyhow, on qemux86-64 the problem seems not reproducible.
Comment 8 Lianhao Lu 2011-09-21 01:32:55 UTC
Built the qemux86 core-image-sato-sdk image on branch 1.1M4, confirmed that this bug is still there.
Comment 9 Jessica 2011-09-21 13:05:42 UTC
Lianhao, what's your host to build the image, mine's 32bit.  I tried today, my agent still able to handle multiple remote debug without seg faulting...
Comment 10 Lianhao Lu 2011-09-21 18:09:39 UTC
My building machine is a 32bit FC13, test machine is 64bit Ubuntu 10.04.
Comment 11 Jessica 2011-09-22 14:18:39 UTC
when tried to create the core file for tcf-agent, I did ulimit -c unlimited, then ulimit -a, then I start tcf-agent (all of this in core-image-sato-sdk for qemux86), not my remote debug and work reliably again, which I haven't able to capture the core file yet, can you try on your end?
Comment 12 Jessica 2011-09-22 14:34:10 UTC
my trace gave me

program terminated with signal 11, Segmentation fault
#0 0x4559b011 in malloc_atfork (sz=100, caller=0x4559afe5) at arena.c:198

and read this link http://bytes.com/topic/c/answers/215865-disadvantages-malloc
Comment 13 Lianhao Lu 2011-09-29 19:09:38 UTC
Created an upstream bug entry at 
https://bugs.eclipse.org/bugs/show_bug.cgi?id=359502
Comment 14 Lianhao Lu 2012-03-22 01:53:27 UTC
wait upstream https://bugs.eclipse.org/bugs/show_bug.cgi?id=359502
Comment 15 Jessica 2012-07-10 17:44:10 UTC
Lianhao and Hongna,

Can you verify if this issue still exist, if not please close the bug
Comment 16 Hongna Xu 2012-07-18 03:03:45 UTC
jessica, i don't meet this issue in 1.3_M2 RC1 testing, git branch: poky/master
git info:  eb0cb7e8234f5d2e5623406e9660be91cf52f65e, but i can't close it cause there is no close states in my options.
Comment 17 Jessica 2012-07-18 17:15:11 UTC
can't reproduce the bug any more
Comment 18 Alexandru Georgescu 2012-07-25 13:46:40 UTC
I tried today with Juno and the latest eclipse-poky. Couldn't reproduce.
Comment 19 Alexandru Georgescu 2012-07-25 13:47:08 UTC
setting the bug to verified-fixed