| Summary: | remote target debugging broken | ||
|---|---|---|---|
| Product: | [Yocto Project Subprojects] ADT | Reporter: | Wolfgang Denk <wd> |
| Component: | adt | Assignee: | Scott Rifenbark <srifenbark> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | major | ||
| Priority: | Medium+ | CC: | alexandru.c.georgescu, alexandru.damian, jessica.zhang, poky.adt.watcher, poky.watcher, srifenbark |
| Version: | 1.3 | ||
| Target Milestone: | 1.1.2 | ||
| Hardware: | All | ||
| OS: | Multiple | ||
| URL: | http://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/commit/?id=863a955f5cf119a38db4950101270bd5a53da027 | ||
| Whiteboard: | 10 April 2013 - IN PROGRESS REVIEW | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
Adding Mark's comments from ML:
>> Trying to do so, we see that it fails. Our current suspicion is that
>> maybe prelinking of the target images and libraries introduces some
>> incompatibility. Is this a reasonable assumption, and if so, is this
>> a problem that should be fixed, or unavoidable for some reason? Or
>> is there a problem with the libraries in OECORE_TARGET_SYSROOT ?
Prelinking should not cause a problem. GDB knows how to translate the debuginfo addresses to the end runtime addresses.
The only thing I've seen in the past that has caused a problem is when the dbg stuff was not installed into the sysroot. GDB will fall back into loading the binaries from the sysroot, which will not contain the debug info causing problems.
It's best if you can set the sysroot to the same filesystem as being executed, but if you can't this is reasonable.
(In reply to comment #1) > > The only thing I've seen in the past that has caused a problem is when the > dbg stuff was not installed into the sysroot. GDB will fall back into > loading the binaries from the sysroot, which will not contain the debug info > causing problems. In such a case I would expect some kind or error (or at least warning) messages; and when a "step" command silently lets the program run until normal termination, something is seriously broken. > It's best if you can set the sysroot to the same filesystem as being > executed, but if you can't this is reasonable. Well, the typical use of gdbserver here is with resource-restricted embedded systems where we have no space for unstripped libraries on the target. And having to create a separate installation of the root file system with debug information for each and every configuration is unacceptable effort considered that OECORE_TARGET_SYSROOT contains the very libraries used on the targets. JEssica, I found that I have errors when trying to Debug an application to a QEMU image. Are these issues related? The error I have in eclipse for a Autotools Hello World project is: Error creating session Could not find platform independent libraries <prefix> Could not find platform independent libraries <prefix> Could not find platform independent libraries <prefix> So I am unclear as to whether we want to go forward with the documentation change suggested here. I can easily make the statement in the manual say the following:
set solib-absolute-prefix ${OECORE_TARGET_SYSROOT}
I need to know if this is okay to go ahead with. Also, I would need to create a new glossary entry for the OECORE_TARGET_SYSROOT variable. I don't know the details behind this variable right now. I will need to get that information.
Scott
Hi Jessica, The same behavior reproduces also on the current danny build for eclipse-plugin: http://autobuilder.yoctoproject.org/pub/nightly/20130226-2/eclipse-plugin/juno/ Alex, Please open a separate bug for your issue, they're not related. I've tried against poky master and everything works. Scott, please update the ADT manual using following steps: 1. setup the cross development environment, toolchain, sysroot (make sure it matches the target rootfs) 2. on remote target end start the gdbserver for a debug session as for example: gdbserver :2345 /usr/bin/hello 3. On the development host under the directory where the to be debugged program is create .gdbinit file and enter: set sysroot pathtoyoursysroot e.g. set sysroot /home/jzhang/sysroot_arm 4. on development host source the cross environment script to setup the correct path and run your crossgdb program, e.g. i586-poky-linux-gdb helloworld. This will start the gdb 5. Inside gdb, connect to remote gdbserver using: target remote remotehost:port e.g. target remote 192.168.7.2:2345 6. setup your break point and walk through the code and other normal gdb actions. E.g. break main continue Jessica, I need some more information to fix the section on remote debugging. This explanation is lengthy so bear with me. First, this topic is split over two manuals: the adt-manual and the dev-manual. The adt-manual describes adequately how to set up the cross-toolchain and access the sysroot and all that. The dev-manual in section 5.15 (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug) describes the actual remote debugging. I need further clarification for the dev-manual steps. I am going to address things according to the steps you put in in your comment: 1. This is covered in the adt-manual 2. Launching the GDB on the target: I think we cover this fine in the section 5.15.1 (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug-launch-gdbserver) section. 3. This step looks new and I will need to add it to the section 5.15.2 (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug-launch-gdb) section. In the manual, this section discusses three things though: building the cross-GDB package either by just getting the binary or having bitbake build it out, making the inferior binaries available by either using an SDK image in the first place or by building a custom directory by reusing tmp/rootfs, and finally launching the host GDB. I am not sure where to put the new step of creating the .gbinit file. Would this be a pre-step to the three things already discussed? I need some guidance on where to place this new requirement of creating the .gbinit file. 4. Running the crossdbg program is the third item discussed in the 5.15.2 section. Currently, there is no mention of sourcing the cross-environment script. Right now that step talks about three things: loading the binary on the host with <target-abi)-gdb rootfs/usr/bin/foo for example, loading the libraries of the inferior binary with set solib-absolute-prefix <path-to-tmp-rootfs>, and then connecting to Gdbserver on the remot with target remote <remote-target-ip-address>:2345 for example. Where exactly do we source the cross-environment script? Are all three of these items still necessary? Or, can you just source the script and then issue the "target" command to connect to the Gdbserver? 5. Your comment here is about connecting to the Gbdserver. I cover that in the third item of "Launching the host GDB" inside section 5.15.2. The command is correct and can stand I think. The only issue is what I brought up in the step 4 above. 6. Stepping through the debug process is fine and is described in section 5.14.2.4 (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug-launch-gdb-using) (In reply to comment #8) Dear Scott, you wrote: > script. Right now that step talks about three things: loading the binary on > the host with <target-abi)-gdb rootfs/usr/bin/foo for example, loading the > libraries of the inferior binary with set solib-absolute-prefix > <path-to-tmp-rootfs>, and then connecting to Gdbserver on the remot with This is where I disagree. There should be no need to keep a copy of the target root file system on the development host. We have all the needed information (binaries and libraries) ready available in the ${OECORE_TARGET_SYSROOT}, so there is really no need at all to maintain yet another copy of this data. PLease also keep in mind that in almost all cases of real embedded systems the libraries instralled in the root file systems will be stripped, so they do not provide the debug information we need, even if we keep a copy of the target root file system on the host. Also, all of this appears to ignore that prelinking appears to play a critical role here - except from Jessica's "should not cause a problem" statement apparently nobody paid any attention to this. Yes, it should not cause problems, that's the theory. Unfortunately what we see is something different. Best regards, Wolfgang Denk Thanks Wolfgang for the further information. I don't think I have enough resolution to document this just yet until we reach bottom on the whole linking issue. If we don't need a copy of the stuff on the host then much of the existing stuff in the section is moot. I think I will wait for Jessica and you to bottom out on the linking stuff. Scott Hi Scott, For step 2 it's launching gdbserver on target not gdb For step 3, we don't need to mention how to build the packages, since the setup cross development environment section (esp. sysroot) should cover it. I think this will cover Wolfgang's concern as well. As to .gdbinit file, it's where the to be debugged program resides, e.g. helloworld. For step 4, just use the directions that I give instead of using the existing ones. You can source your environment setup file any location, since it'll setup your environment to be able to access the cross debugger, etc. The old content is quite stale and since we heavily using sysroot for cross development which simplified things a log, so my steps are the mose up-to-date version, just stick to them. Wolfgang, I noticed the bug was against 1.1.2, I've tested the steps using latest master. So I think the problem that you've experienced should have been resolved and with the updated steps in doc, people shouldn't experience it any more. THanks, Jessica Jessica, For step three (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug-launch-gdb-buildcross), which is titled "Building the Cross-GDB Package", should we remove the section entirely, or just get rid of the part that talks about building it out using bitbake? Scott, Step 3 should replace the current 5.15.2.2 and 5.15.2.3. 5.15.2.1 is still valid Jessica, Here is the section "Lauching GDB on the Host Computer (http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug-launch-gdb) with the first attempt at changes. The flow is now basically this: 1. Build the Cross-GDB Package 2. Create the GDB Initialization File 3. Launch the Host GDB 4. Use the Debugger I completely removed the existing 5.15.2.2 and 5.15.2.3 sections. There seemed to be quite a bit of detail in those deleted sections. Please look this section over and see if it works. Thanks, Scott Scott, It doesn't look quite right. For 5.16.2.3, that line needs to go into the .gdbinit file instead of a separate command. For launch host gdb, since we're using .gdbinit file, I'd suggest we switch to the directory where the binary resides and start gdb there as my example: i586-poky-linux-gdb helloworld You miss to mention step 5 that in my comments, you need to connect to the remote gdbserver to create the debugging session. Also, in the section there're 2 binaries used inferio and foo. I'd rather us to stick with one since we start the server for one binary and launch host gdb for another, which won't work. THanks, Jessica Jessica, Here is a link to the rewritten section: http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug There might be issues still as I am unsure as to the binaries involved. I pretty much eliminated references to the "inferior" binary and stuck with using "helloworld". What I don't understand is if there are two binaries involved: one on the host and one on the target. Maybe you can explain that to me. I get the overall process of: 1. Setting up the debugging environment (toolchains, sysroot, etc.) 2. Launching Gdbserver on the target and getting it to listen for the host through some port. 3. Launching the GDB server on the host computer 4. Connect to the remote GDB Server from the target and making the ports match up. 5. Then using the debugger Is there only one binary and if so, where does it live? Let me know how this attempt at the section is. Thanks, Scott Scott, We're getting there but still 2 things to make the examples more clearer, and I realize that in my steps I didn't provide the consistent data: 1. the binary need to be helloworld instead of hello on target 2. I'd like us to stick with arm architecture since it's being used in building cross-gdb and sysroot. So for the example of bring up host side cross gdb, change to: arm-poky-linux-gnuabi-gdb helloworld Thanks, Jessica Jessica, I updated the section with these two changes: http://www.yoctoproject.org/docs/1.4/dev-manual/dev-manual.html#platdev-gdb-remotedebug If I don't hear from you then I will consider it good. Thanks, Scott I've reviewed the latest version, and everything looks good |
The actual bug that triggers this entry is that remote target debugging using the cross GDB on the host and gdbserver on the target does not work; attempts to single-step (using sequences of step-step-step or step-next-step) will raise errors or the program will simply run until normal termination. The problem goes away, if - you have the libraries with symbols installed in the target root file system, and - you use a "set sysroot" or "set solib-absolute-prefix" command to point GDB to the root file system Trying to use the libraries in ${OECORE_TARGET_SYSROOT} as reference for GDB does not work with the default file systems in Yocto. It turns out that the reason for this behaviour is prelinking. As soon as prelinking happens, the target libraries become incompatible (as far as GDB use is concerned) to the libraries in ${OECORE_TARGET_SYSROOT}. There are several problems in different areas here: 1. Documentation: The "Yocto Project Development Manual" says in section "5.14.2.3. Launch the Host GDB" that one should use $ set solib-absolute-prefix /path/to/tmp/rootfs [see http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#platdev-gdb-remotedebug-launch-gdb ] I think this is at least not optimal; pointing to (a copy of) the target root file system (where we usually have stripped libraries) requires special steps to be performed to make debug symbols available, and again we will suffer from the fact that libraries in this copy of the rootfs (which are NOT prelinked when you install these from the respective packages) will be incompatible with the target rootfs. Instead, we should be able to use the libraries in ${OECORE_TARGET_SYSROOT}. ==> Documentation should be changed to point to ${OECORE_TARGET_SYSROOT}. 2. Code does not match documentation: The eclipse plugin generates a .gdbinit script which uses ${OECORE_TARGET_SYSROOT} as I suggest here - this is inconsistent with the documentation in the Yocto Project Development Manual". ==> Documentation should be fixed. 3. Prelinking makes debugging impossible It is not clear to me if this is a problem in the cross-prelinker used when building the rootfs images, or problem with prelink in general. In any case, prelinking should not cause such debug issues, otherwise it cannot be used in any systems where code may need debugging. ==> Prelinking should be fixed not to mess with GDB debugging.