Bug 9481

The main difference than before is:

4. On host side, create symlink /usr/src/debug to
   ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the
   location of sources is /usr/src/debug.

$ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug

8. Use path ${STAGING_DIR_HOST} as the system root for
   the program being debugged. Any absolute shared
   library paths will be prefixed with path;

(gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86

//Hongxu

> 4. On host side, create symlink /usr/src/debug to
>    ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the
>    location of sources is /usr/src/debug.
> 
> $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug
> 

As Mark Hatle suggested, we use a better way to set the substitution
of source path, here is the update of 4.

4. On host side, Set a substitution rule for finding source files.
   Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}

(gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux

//Hongxu

Changed in V2:
- Move 4) to 8)

Example:

1. Build a image with gdbserver and gzip installed. The gzip
   is the debug binary. Append the following to local.conf
...
IMAGE_INSTALL_append = " gdbserver gzip"
...

$ bitbake core-image-sato

2. Build gdb-cross for remote debug on host

$ bitbake gdb-cross-i586

3. start qemu

$ runqemu qemux86 core-image-sato

4. On target side, ipaddr is 128.224.163.187,
   run gdbserver to debug "gzip -h"

root@qemux86-64:~# gdbserver localhost:1024 gzip -h
Process gzip created; pid = 530
Listening on port 1024

5. On host side, enter gzip's devshell
$ bitbake -cdevshell gzip

6. On host side, run gdb-cross to remote debug gzip

gzip-1.6# i586-poky-linux-gdb ../build/gzip

7. Use path ${STAGING_DIR_HOST} as the system root for
   the program being debugged. Any absolute shared
   library paths will be prefixed with path;

(gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86

8. On host side, set a substitution rule for finding source files.
   Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}

(gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux

9. Start remote target, set break point at help()

(gdb) target remote 128.224.163.187:1024
(gdb) break gzip.c:325
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000000000401ec1 in main at /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325
(gdb) c
Continuing.

Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465
465                 help(); do_exit(OK); break;
(gdb) list
460             case 'd':
461                 decompress = 1; break;
462             case 'f':
463                 force++; break;
464             case 'h': case 'H':
465                 help(); do_exit(OK); break;
466             case 'k':
467                 keep = 1; break;
468             case 'l':
469                 list = decompress = to_stdout = 1; break;
(gdb) c
Continuing.
[Inferior 1 (process 1246) exited normally]


10. On target side, get output of "gzip -h"
[snip]
Remote debugging from host 128.224.153.74
Usage: gzip [OPTION]... [FILE]...
Compress or uncompress FILEs (by default, compress FILES in-place).

Mandatory arguments to long options are mandatory for short options too.

  -c, --stdout      write on standard output, keep original files unchanged
  -d, --decompress  decompress
  -f, --force       force overwrite of output file and compress links
  -h, --help        give this help
  -k, --keep        keep (don't delete) input files
  -l, --list        list compressed file contents
  -L, --license     display software license
  -n, --no-name     do not save or restore the original name and time stamp
  -N, --name        save or restore the original name and time stamp
  -q, --quiet       suppress all warnings
  -r, --recursive   operate recursively on directories
  -S, --suffix=SUF  use suffix SUF on compressed files
  -t, --test        test compressed file integrity
  -v, --verbose     verbose mode
  -V, --version     display version number
  -1, --fast        compress faster
  -9, --best        compress better

With no FILE, or when FILE is -, read standard input.

Report bugs to <bug-gzip@gnu.org>.

Child exited with status 0
GDBserver exiting
[snip]

//Hongxu

So the main difference than before is:

7. Use path ${STAGING_DIR_HOST} as the system root for
   the program being debugged. Any absolute shared
   library paths will be prefixed with path;

(gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86

8. On host side, set a substitution rule for finding source files.
   Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}

(gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux

//Hongxu

In http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#platdev-gdb-remotedebug

5.23.3.2. Create the GDB Initialization File and Point to Your Root Filesystem¶

Aside from the GDB cross-binary, you also need a GDB initialization file in the same top directory in which your binary resides. When you start GDB on your host development system, GDB finds this initialization file and executes all the commands within. For information on the .gdbinit, see "Debugging with GDB", which is maintained by sourceware.org.

You need to add a statement in the ~/.gdbinit file that points to your root filesystem. Here is an example that points to the root filesystem for target device:

     set sysroot <value of ${STAGING_DIR_HOST}>
     set substitute-path /usr/src/debug <value of ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}>

I'm writing up a complete set of steps.  It will take me a few minutes to perform a build and verify the steps are correct.

My concern with the previous comments is that these steps are very specific on a use-case of attaching the debugger to a work dir.  This is not and should not be the typical debugging way a user debugs a target.  (It can be an alternative use-case, but shouldn't be the primary.)

(In reply to comment #6)
> I'm writing up a complete set of steps.  It will take me a few minutes to
> perform a build and verify the steps are correct.
> 
> My concern with the previous comments is that these steps are very specific
> on a use-case of attaching the debugger to a work dir.  This is not and
> should not be the typical debugging way a user debugs a target.  (It can be
> an alternative use-case, but shouldn't be the primary.)

Agree

//Hongxu

(There is currently a bug in master where the IMAGE_FSTYPES_DEBUGFS settings do not work.  This will be fixed in the release, or specific limitations will be noted.)

5.23:  I would remove the note.  It’s simply going to be confusing to someone.  Without the src’s in the -dbg packages things become undebuggable.  The only advantage is that you can inspect variables and get backtraces — but you have no correlation to the actual source used to build the binary.

At this stage of the documentation, I wouldn’t want to confuse someone with these types of options, as they’re corner cases — and should not be standard configurations.

The rest of the parts of 5.23.* should be modified to match the items below.

1. Configure your build system to construct the companion debug filesystem

In the local.conf:

IMAGE_GEN_DEBUGFS = '1'
IMAGE_FSTYPES_DEBUGFS = 'tar.bz2'

The options above will cause the system to generate a special companion filesystem fragment, that contains the matching source and debug symbols to your deployable filesystem.  It does this by looking at what is in the deployed filesystem, and pulling the corresponding -dbg packages.

The companion debug filesystem is NOT a complete filesystem, but only contains the debug fragements.  It must be combined with the full filesystem for debugging.  (Later steps will show this.)

2. Configure the system to include gdbserver in the target filesystem

In the local.conf (or an image recipe):

IMAGE_INSTALL_append = “ gdbserver"

3. Build the environment

Construct the image (and companion Debug Filesystem)
   bitbake <image>

Build the cross GDB component and make it available for debugging

 Build the SDK that matches the image (this is best for a production build for later debugging, especially during long term maintenance)
   bitbake -c populate_sdk <image>

   or 

 Build the minimal toolchain components that match the target (this is smaller then a typical SDK and only contains a minimal set of components to build simple test applications, as well as run the debugger)
   bitbake meta-toolchain

   or

 Build gdb itself within the build system (this produces a temporary copy of cross-gdb that can be used for debugging during development.  It is the quickest approach, but the other methods are better for long term maintenance strategies.)
   bitbake gdb-cross-<arch>

 (Note: If you run bitbake gdb-cross, the system will give you a suggestion like gdb-cross-i586, the suggestion is likely the actual name you want to use.)

4. Setup your environment

4.1 Setup the debugfs

$ mkdir debugfs
$ cd debugfs
$ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>.rootfs.tar.bz2
$ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>-dbg.rootfs.tar.bz2

4.2 Setup GDB

Install the SDK (if you built one) and source the correct environment file.  This will put it in your path.

if using the build system gdb will be in <build>/<tmp>/sysroots/<host>/usr/bin/<arch>/<arch>-gdb

5. Boot the target

For QEMU, <link to runqemu docs…>

(verify that your host can access the target via TCP)

6. Debug a program

6.1 Run gdbserver on the target, such as:

(We’ll debug gzip in this example, see gdbserver docs for additional options)

root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help

6.2 On the host, run gdb, configure it and connect to the target:

$ cd <directory holding the debugfs dir>
$ <arch>-gdb

(gdb) set sysroot debugfs
(gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
(gdb) target remote <ip of target>:1234

At this point everything should automatically load (matching binaries, symbols and headers.)

Note: the gdb ‘set’ commands above can be placed into the users ~/.gdbinit   GDB will automatically run whatever commands are in that file when it is started.


7. Deploying WITHOUT a full image rebuild

In many cases, during development you want a quick method to deploy a new binary to the target and debug it, without waiting for a full image build.

One approach to this is to build the component you want to debug.  Then directly copy the executable to BOTH the target and the host ‘debugfs’.  If the binary is processed through the debug splitting in OE, you should also copy the debug items (the .debug contents and corresponding /usr/src/debug) from the work dir.

Such as:

$ bitbake bash
$ bitbake -c devshell bash
$ cd ..
$ scp packages-split/bash/bin/bash <target>:/bin/bash
$ cp -a packages-split/bash-dbg/* <path>/debugfs

Example in step 7 was missing a copy...

Should be:

7. Deploying WITHOUT a full image rebuild

In many cases, during development you want a quick method to deploy a new binary to the target and debug it, without waiting for a full image build.

One approach to this is to build the component you want to debug.  Then directly copy the executable to BOTH the target and the host ‘debugfs’.  If the binary is processed through the debug splitting in OE, you should also copy the debug items (the .debug contents and corresponding /usr/src/debug) from the work dir.

Such as:

$ bitbake bash
$ bitbake -c devshell bash
$ cd ..
$ scp packages-split/bash/bin/bash <target>:/bin/bash
$ cp packages-split/bash/bin/bash <path>/debugfs/bin/bash
$ cp -r packages-split/bash-dbg/* <path>/debugfs

added myself to the CC list for the doc component.

Scott

Mark, 

I am looking at the section in the dev-manual at http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#platdev-gdb-remotedebug and trying to map the steps to what is there in the 5.24.* subsections.  They don't map that well.  Should I replace the entire existing 5.24 section with the steps you have in your comments here?

Thanks, 
Scott

Setting to NEEDINFO.  Please see my last comment.

Scott

All of the steps for building the SDK and/or cross-gdb are still applicable.  The thing that changes is primarily around the notes to "be sure to include the -dbg" packages in your image.  That is simply not necessary.

Instead if you use the steps from my previous comment you can produce a release image, and a corresponding debug symbol image.

Then on the target you deploy the release image, on the host you can use the debug symbol image with the cross debugger.

(In reply to comment #8)
> (There is currently a bug in master where the IMAGE_FSTYPES_DEBUGFS settings
> do not work.  This will be fixed in the release, or specific limitations
> will be noted.)
> 
> 5.23:  I would remove the note.  It’s simply going to be confusing to
> someone.  Without the src’s in the -dbg packages things become undebuggable.
> The only advantage is that you can inspect variables and get backtraces —
> but you have no correlation to the actual source used to build the binary.
> 
> At this stage of the documentation, I wouldn’t want to confuse someone with
> these types of options, as they’re corner cases — and should not be standard
> configurations.

I am assuming that the section under question is 5.24.  I think since you had added these comments a new sub-section was added.  So, I am removing the "note" at the end of the section.  I am leaving the "tip". 

> 
> The rest of the parts of 5.23.* should be modified to match the items below.

I am removing the sub-section scheme completely and replacing it with a numbered list based on your steps here.

> 
> 1. Configure your build system to construct the companion debug filesystem
> 
> In the local.conf:
> 
> IMAGE_GEN_DEBUGFS = '1'
> IMAGE_FSTYPES_DEBUGFS = 'tar.bz2'
> 
> The options above will cause the system to generate a special companion
> filesystem fragment, that contains the matching source and debug symbols to
> your deployable filesystem.  It does this by looking at what is in the
> deployed filesystem, and pulling the corresponding -dbg packages.
> 
> The companion debug filesystem is NOT a complete filesystem, but only
> contains the debug fragements.  It must be combined with the full filesystem
> for debugging.  (Later steps will show this.)
> 
> 2. Configure the system to include gdbserver in the target filesystem
> 
> In the local.conf (or an image recipe):
> 
> IMAGE_INSTALL_append = “ gdbserver"
> 
> 3. Build the environment
> 
> Construct the image (and companion Debug Filesystem)
>    bitbake <image>
> 
> Build the cross GDB component and make it available for debugging
> 
>  Build the SDK that matches the image (this is best for a production build
> for later debugging, especially during long term maintenance)
>    bitbake -c populate_sdk <image>
> 
>    or 
> 
>  Build the minimal toolchain components that match the target (this is
> smaller then a typical SDK and only contains a minimal set of components to
> build simple test applications, as well as run the debugger)
>    bitbake meta-toolchain
> 
>    or
> 
>  Build gdb itself within the build system (this produces a temporary copy of
> cross-gdb that can be used for debugging during development.  It is the
> quickest approach, but the other methods are better for long term
> maintenance strategies.)
>    bitbake gdb-cross-<arch>
> 
>  (Note: If you run bitbake gdb-cross, the system will give you a suggestion
> like gdb-cross-i586, the suggestion is likely the actual name you want to
> use.)
> 
> 4. Setup your environment
> 
> > 4.1 Setup the debugfs
> 
> $ mkdir debugfs
> $ cd debugfs
> $ tar xvfj <build
> dir>/tmp-glibc/deploy/images/<machine>/<image>.rootfs.tar.bz2
> $ tar xvfj <build
> dir>/tmp-glibc/deploy/images/<machine>/<image>-dbg.rootfs.tar.bz2
> 
4.2 Setup GDB
> 
> Install the SDK (if you built one) and source the correct environment file. 
> This will put it in your path.
> 
> if using the build system gdb will be in
> <build>/<tmp>/sysroots/<host>/usr/bin/<arch>/<arch>-gdb
> 
> 5. Boot the target
> 
> For QEMU, <link to runqemu docs…>
> 
> (verify that your host can access the target via TCP)
> 
> 6. Debug a program
> 
> 6.1 Run gdbserver on the target, such as:
> 
> (We’ll debug gzip in this example, see gdbserver docs for additional options)
> 
> root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
> 
> 6.2 On the host, run gdb, configure it and connect to the target:
> 
> $ cd <directory holding the debugfs dir>
> $ <arch>-gdb
> 
> (gdb) set sysroot debugfs
> (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
> (gdb) target remote <ip of target>:1234
> 
> At this point everything should automatically load (matching binaries,
> symbols and headers.)
> 
> Note: the gdb ‘set’ commands above can be placed into the users ~/.gdbinit  
> GDB will automatically run whatever commands are in that file when it is
> started.
> 
> 
> 7. Deploying WITHOUT a full image rebuild
> 
> In many cases, during development you want a quick method to deploy a new
> binary to the target and debug it, without waiting for a full image build.
> 
> One approach to this is to build the component you want to debug.  Then
> directly copy the executable to BOTH the target and the host ‘debugfs’.  If
> the binary is processed through the debug splitting in OE, you should also
> copy the debug items (the .debug contents and corresponding /usr/src/debug)
> from the work dir.
> 
> Such as:
> 
> $ bitbake bash
> $ bitbake -c devshell bash
> $ cd ..
> $ scp packages-split/bash/bin/bash <target>:/bin/bash
> $ cp -a packages-split/bash-dbg/* <path>/debugfs

You can see the new section at http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#platdev-gdb-remotedebug.  It needs to be reviewed as I did some rewriting and guessing in some spots. 

Thanks,
Scott

It has been quite some time since this change was made for the docs.  I have yet to get an "okay" on the solution. 

Thanks, 
Scott

Sorry thought I had responded previously.

I've reviewed the new section and it appears to be correct and functional to me.

Mark, 

Thanks for getting back to me.  I am marking the bug RESOLVED and setting the Doc flag to "done."

Scott