9481 2016-04-19 07:23:26 +0000 Add example to "5.23. Debugging With the GNU Project Debugger (GDB) Remotely" 2017-01-18 23:35:47 +0000 1 1 9 Documentation Development Manual development unspecified x86 Multiple RESOLVED FIXED 18 Jan 2017: RESOLVED Medium+ enhancement 2.3 M2 1 hongxu.jia mark.hatle jose.perez.carranza liezhi.yang mark.hatle patrick.ohly richard.purdie ross.burton schnitzeltony srifenbark francisco.j.pedraza.gonzalez --- --- Done (doc changes complete) oldest_to_newest 61370 0 hongxu.jia 2016-04-19 07:23:26 +0000 Since -fdebug-prefix-map was added to DEBUG_FLAGS, previously remote debug could not work correctly, so we add an example to explain how to do remote debug. Example: 1. Build a image with gdbserver and gzip installed. The gzip is the debug binary. Append the following to local.conf ... IMAGE_INSTALL_append = " gdbserver gzip" ... $ bitbake core-image-sato 2. Build gdb-cross for remote debug on host $ bitbake gdb-cross-i586 3. start qemu $ runqemu qemux86 core-image-sato 4. On host side, create symlink /usr/src/debug to ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the location of sources is /usr/src/debug. $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug 5. On target side, ipaddr is 128.224.163.187, run gdbserver to debug "gzip -h" root@qemux86-64:~# gdbserver localhost:1024 gzip -h Process gzip created; pid = 530 Listening on port 1024 6. On host side, enter gzip's devshell $ bitbake -cdevshell gzip 7. On host side, run gdb-cross to remote debug gzip gzip-1.6# i586-poky-linux-gdb ../build/gzip 8. Use path ${STAGING_DIR_HOST} as the system root for the program being debugged. Any absolute shared library paths will be prefixed with path; (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86 9. Start remote target, set break point at help() (gdb) target remote 128.224.163.187:1024 (gdb) break gzip.c:325 (gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y 0x0000000000401ec1 in main at /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325 (gdb) c Continuing. Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465 465 help(); do_exit(OK); break; (gdb) list 460 case 'd': 461 decompress = 1; break; 462 case 'f': 463 force++; break; 464 case 'h': case 'H': 465 help(); do_exit(OK); break; 466 case 'k': 467 keep = 1; break; 468 case 'l': 469 list = decompress = to_stdout = 1; break; (gdb) c Continuing. [Inferior 1 (process 1246) exited normally] 10. On target side, get output of "gzip -h" [snip] Remote debugging from host 128.224.153.74 Usage: gzip [OPTION]... [FILE]... Compress or uncompress FILEs (by default, compress FILES in-place). Mandatory arguments to long options are mandatory for short options too. -c, --stdout write on standard output, keep original files unchanged -d, --decompress decompress -f, --force force overwrite of output file and compress links -h, --help give this help -k, --keep keep (don't delete) input files -l, --list list compressed file contents -L, --license display software license -n, --no-name do not save or restore the original name and time stamp -N, --name save or restore the original name and time stamp -q, --quiet suppress all warnings -r, --recursive operate recursively on directories -S, --suffix=SUF use suffix SUF on compressed files -t, --test test compressed file integrity -v, --verbose verbose mode -V, --version display version number -1, --fast compress faster -9, --best compress better With no FILE, or when FILE is -, read standard input. Report bugs to <bug-gzip@gnu.org>. Child exited with status 0 GDBserver exiting [snip] //Hongxu 61371 1 hongxu.jia 2016-04-19 08:02:56 +0000 The main difference than before is: 4. On host side, create symlink /usr/src/debug to ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the location of sources is /usr/src/debug. $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug 8. Use path ${STAGING_DIR_HOST} as the system root for the program being debugged. Any absolute shared library paths will be prefixed with path; (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86 //Hongxu 61395 2 hongxu.jia 2016-04-19 13:03:09 +0000 > 4. On host side, create symlink /usr/src/debug to > ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the > location of sources is /usr/src/debug. > > $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug > As Mark Hatle suggested, we use a better way to set the substitution of source path, here is the update of 4. 4. On host side, Set a substitution rule for finding source files. Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} (gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux //Hongxu 61396 3 hongxu.jia 2016-04-19 13:06:50 +0000 Changed in V2: - Move 4) to 8) Example: 1. Build a image with gdbserver and gzip installed. The gzip is the debug binary. Append the following to local.conf ... IMAGE_INSTALL_append = " gdbserver gzip" ... $ bitbake core-image-sato 2. Build gdb-cross for remote debug on host $ bitbake gdb-cross-i586 3. start qemu $ runqemu qemux86 core-image-sato 4. On target side, ipaddr is 128.224.163.187, run gdbserver to debug "gzip -h" root@qemux86-64:~# gdbserver localhost:1024 gzip -h Process gzip created; pid = 530 Listening on port 1024 5. On host side, enter gzip's devshell $ bitbake -cdevshell gzip 6. On host side, run gdb-cross to remote debug gzip gzip-1.6# i586-poky-linux-gdb ../build/gzip 7. Use path ${STAGING_DIR_HOST} as the system root for the program being debugged. Any absolute shared library paths will be prefixed with path; (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86 8. On host side, set a substitution rule for finding source files. Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} (gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux 9. Start remote target, set break point at help() (gdb) target remote 128.224.163.187:1024 (gdb) break gzip.c:325 (gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y 0x0000000000401ec1 in main at /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325 (gdb) c Continuing. Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465 465 help(); do_exit(OK); break; (gdb) list 460 case 'd': 461 decompress = 1; break; 462 case 'f': 463 force++; break; 464 case 'h': case 'H': 465 help(); do_exit(OK); break; 466 case 'k': 467 keep = 1; break; 468 case 'l': 469 list = decompress = to_stdout = 1; break; (gdb) c Continuing. [Inferior 1 (process 1246) exited normally] 10. On target side, get output of "gzip -h" [snip] Remote debugging from host 128.224.153.74 Usage: gzip [OPTION]... [FILE]... Compress or uncompress FILEs (by default, compress FILES in-place). Mandatory arguments to long options are mandatory for short options too. -c, --stdout write on standard output, keep original files unchanged -d, --decompress decompress -f, --force force overwrite of output file and compress links -h, --help give this help -k, --keep keep (don't delete) input files -l, --list list compressed file contents -L, --license display software license -n, --no-name do not save or restore the original name and time stamp -N, --name save or restore the original name and time stamp -q, --quiet suppress all warnings -r, --recursive operate recursively on directories -S, --suffix=SUF use suffix SUF on compressed files -t, --test test compressed file integrity -v, --verbose verbose mode -V, --version display version number -1, --fast compress faster -9, --best compress better With no FILE, or when FILE is -, read standard input. Report bugs to <bug-gzip@gnu.org>. Child exited with status 0 GDBserver exiting [snip] //Hongxu 61397 4 hongxu.jia 2016-04-19 13:15:28 +0000 So the main difference than before is: 7. Use path ${STAGING_DIR_HOST} as the system root for the program being debugged. Any absolute shared library paths will be prefixed with path; (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86 8. On host side, set a substitution rule for finding source files. Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} (gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux //Hongxu 61398 5 hongxu.jia 2016-04-19 13:18:13 +0000 In http://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#platdev-gdb-remotedebug 5.23.3.2. Create the GDB Initialization File and Point to Your Root Filesystem¶ Aside from the GDB cross-binary, you also need a GDB initialization file in the same top directory in which your binary resides. When you start GDB on your host development system, GDB finds this initialization file and executes all the commands within. For information on the .gdbinit, see "Debugging with GDB", which is maintained by sourceware.org. You need to add a statement in the ~/.gdbinit file that points to your root filesystem. Here is an example that points to the root filesystem for target device: set sysroot <value of ${STAGING_DIR_HOST}> set substitute-path /usr/src/debug <value of ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}> 61399 6 mark.hatle 2016-04-19 13:22:32 +0000 I'm writing up a complete set of steps. It will take me a few minutes to perform a build and verify the steps are correct. My concern with the previous comments is that these steps are very specific on a use-case of attaching the debugger to a work dir. This is not and should not be the typical debugging way a user debugs a target. (It can be an alternative use-case, but shouldn't be the primary.) 61402 7 hongxu.jia 2016-04-19 13:37:03 +0000 (In reply to comment #6) > I'm writing up a complete set of steps. It will take me a few minutes to > perform a build and verify the steps are correct. > > My concern with the previous comments is that these steps are very specific > on a use-case of attaching the debugger to a work dir. This is not and > should not be the typical debugging way a user debugs a target. (It can be > an alternative use-case, but shouldn't be the primary.) Agree //Hongxu 61406 8 mark.hatle 2016-04-19 14:43:24 +0000 (There is currently a bug in master where the IMAGE_FSTYPES_DEBUGFS settings do not work. This will be fixed in the release, or specific limitations will be noted.) 5.23: I would remove the note. It’s simply going to be confusing to someone. Without the src’s in the -dbg packages things become undebuggable. The only advantage is that you can inspect variables and get backtraces — but you have no correlation to the actual source used to build the binary. At this stage of the documentation, I wouldn’t want to confuse someone with these types of options, as they’re corner cases — and should not be standard configurations. The rest of the parts of 5.23.* should be modified to match the items below. 1. Configure your build system to construct the companion debug filesystem In the local.conf: IMAGE_GEN_DEBUGFS = '1' IMAGE_FSTYPES_DEBUGFS = 'tar.bz2' The options above will cause the system to generate a special companion filesystem fragment, that contains the matching source and debug symbols to your deployable filesystem. It does this by looking at what is in the deployed filesystem, and pulling the corresponding -dbg packages. The companion debug filesystem is NOT a complete filesystem, but only contains the debug fragements. It must be combined with the full filesystem for debugging. (Later steps will show this.) 2. Configure the system to include gdbserver in the target filesystem In the local.conf (or an image recipe): IMAGE_INSTALL_append = “ gdbserver" 3. Build the environment Construct the image (and companion Debug Filesystem) bitbake <image> Build the cross GDB component and make it available for debugging Build the SDK that matches the image (this is best for a production build for later debugging, especially during long term maintenance) bitbake -c populate_sdk <image> or Build the minimal toolchain components that match the target (this is smaller then a typical SDK and only contains a minimal set of components to build simple test applications, as well as run the debugger) bitbake meta-toolchain or Build gdb itself within the build system (this produces a temporary copy of cross-gdb that can be used for debugging during development. It is the quickest approach, but the other methods are better for long term maintenance strategies.) bitbake gdb-cross-<arch> (Note: If you run bitbake gdb-cross, the system will give you a suggestion like gdb-cross-i586, the suggestion is likely the actual name you want to use.) 4. Setup your environment 4.1 Setup the debugfs $ mkdir debugfs $ cd debugfs $ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>.rootfs.tar.bz2 $ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>-dbg.rootfs.tar.bz2 4.2 Setup GDB Install the SDK (if you built one) and source the correct environment file. This will put it in your path. if using the build system gdb will be in <build>/<tmp>/sysroots/<host>/usr/bin/<arch>/<arch>-gdb 5. Boot the target For QEMU, <link to runqemu docs…> (verify that your host can access the target via TCP) 6. Debug a program 6.1 Run gdbserver on the target, such as: (We’ll debug gzip in this example, see gdbserver docs for additional options) root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help 6.2 On the host, run gdb, configure it and connect to the target: $ cd <directory holding the debugfs dir> $ <arch>-gdb (gdb) set sysroot debugfs (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug (gdb) target remote <ip of target>:1234 At this point everything should automatically load (matching binaries, symbols and headers.) Note: the gdb ‘set’ commands above can be placed into the users ~/.gdbinit GDB will automatically run whatever commands are in that file when it is started. 7. Deploying WITHOUT a full image rebuild In many cases, during development you want a quick method to deploy a new binary to the target and debug it, without waiting for a full image build. One approach to this is to build the component you want to debug. Then directly copy the executable to BOTH the target and the host ‘debugfs’. If the binary is processed through the debug splitting in OE, you should also copy the debug items (the .debug contents and corresponding /usr/src/debug) from the work dir. Such as: $ bitbake bash $ bitbake -c devshell bash $ cd .. $ scp packages-split/bash/bin/bash <target>:/bin/bash $ cp -a packages-split/bash-dbg/* <path>/debugfs 61407 9 mark.hatle 2016-04-19 14:48:51 +0000 Example in step 7 was missing a copy... Should be: 7. Deploying WITHOUT a full image rebuild In many cases, during development you want a quick method to deploy a new binary to the target and debug it, without waiting for a full image build. One approach to this is to build the component you want to debug. Then directly copy the executable to BOTH the target and the host ‘debugfs’. If the binary is processed through the debug splitting in OE, you should also copy the debug items (the .debug contents and corresponding /usr/src/debug) from the work dir. Such as: $ bitbake bash $ bitbake -c devshell bash $ cd .. $ scp packages-split/bash/bin/bash <target>:/bin/bash $ cp packages-split/bash/bin/bash <path>/debugfs/bin/bash $ cp -r packages-split/bash-dbg/* <path>/debugfs 63156 10 srifenbark 2016-06-20 15:44:16 +0000 added myself to the CC list for the doc component. Scott 63484 11 srifenbark 2016-06-29 20:17:33 +0000 Mark, I am looking at the section in the dev-manual at http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#platdev-gdb-remotedebug and trying to map the steps to what is there in the 5.24.* subsections. They don't map that well. Should I replace the entire existing 5.24 section with the steps you have in your comments here? Thanks, Scott 66595 12 srifenbark 2016-09-26 16:43:46 +0000 Setting to NEEDINFO. Please see my last comment. Scott 67124 13 mark.hatle 2016-10-11 07:31:35 +0000 All of the steps for building the SDK and/or cross-gdb are still applicable. The thing that changes is primarily around the notes to "be sure to include the -dbg" packages in your image. That is simply not necessary. Instead if you use the steps from my previous comment you can produce a release image, and a corresponding debug symbol image. Then on the target you deploy the release image, on the host you can use the debug symbol image with the cross debugger. 67145 14 srifenbark 2016-10-11 16:47:29 +0000 (In reply to comment #8) > (There is currently a bug in master where the IMAGE_FSTYPES_DEBUGFS settings > do not work. This will be fixed in the release, or specific limitations > will be noted.) > > 5.23: I would remove the note. It’s simply going to be confusing to > someone. Without the src’s in the -dbg packages things become undebuggable. > The only advantage is that you can inspect variables and get backtraces — > but you have no correlation to the actual source used to build the binary. > > At this stage of the documentation, I wouldn’t want to confuse someone with > these types of options, as they’re corner cases — and should not be standard > configurations. I am assuming that the section under question is 5.24. I think since you had added these comments a new sub-section was added. So, I am removing the "note" at the end of the section. I am leaving the "tip". > > The rest of the parts of 5.23.* should be modified to match the items below. I am removing the sub-section scheme completely and replacing it with a numbered list based on your steps here. > > 1. Configure your build system to construct the companion debug filesystem > > In the local.conf: > > IMAGE_GEN_DEBUGFS = '1' > IMAGE_FSTYPES_DEBUGFS = 'tar.bz2' > > The options above will cause the system to generate a special companion > filesystem fragment, that contains the matching source and debug symbols to > your deployable filesystem. It does this by looking at what is in the > deployed filesystem, and pulling the corresponding -dbg packages. > > The companion debug filesystem is NOT a complete filesystem, but only > contains the debug fragements. It must be combined with the full filesystem > for debugging. (Later steps will show this.) > > 2. Configure the system to include gdbserver in the target filesystem > > In the local.conf (or an image recipe): > > IMAGE_INSTALL_append = “ gdbserver" > > 3. Build the environment > > Construct the image (and companion Debug Filesystem) > bitbake <image> > > Build the cross GDB component and make it available for debugging > > Build the SDK that matches the image (this is best for a production build > for later debugging, especially during long term maintenance) > bitbake -c populate_sdk <image> > > or > > Build the minimal toolchain components that match the target (this is > smaller then a typical SDK and only contains a minimal set of components to > build simple test applications, as well as run the debugger) > bitbake meta-toolchain > > or > > Build gdb itself within the build system (this produces a temporary copy of > cross-gdb that can be used for debugging during development. It is the > quickest approach, but the other methods are better for long term > maintenance strategies.) > bitbake gdb-cross-<arch> > > (Note: If you run bitbake gdb-cross, the system will give you a suggestion > like gdb-cross-i586, the suggestion is likely the actual name you want to > use.) > > 4. Setup your environment > > > 4.1 Setup the debugfs > > $ mkdir debugfs > $ cd debugfs > $ tar xvfj <build > dir>/tmp-glibc/deploy/images/<machine>/<image>.rootfs.tar.bz2 > $ tar xvfj <build > dir>/tmp-glibc/deploy/images/<machine>/<image>-dbg.rootfs.tar.bz2 > 4.2 Setup GDB > > Install the SDK (if you built one) and source the correct environment file. > This will put it in your path. > > if using the build system gdb will be in > <build>/<tmp>/sysroots/<host>/usr/bin/<arch>/<arch>-gdb > > 5. Boot the target > > For QEMU, <link to runqemu docs…> > > (verify that your host can access the target via TCP) > > 6. Debug a program > > 6.1 Run gdbserver on the target, such as: > > (We’ll debug gzip in this example, see gdbserver docs for additional options) > > root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help > > 6.2 On the host, run gdb, configure it and connect to the target: > > $ cd <directory holding the debugfs dir> > $ <arch>-gdb > > (gdb) set sysroot debugfs > (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug > (gdb) target remote <ip of target>:1234 > > At this point everything should automatically load (matching binaries, > symbols and headers.) > > Note: the gdb ‘set’ commands above can be placed into the users ~/.gdbinit > GDB will automatically run whatever commands are in that file when it is > started. > > > 7. Deploying WITHOUT a full image rebuild > > In many cases, during development you want a quick method to deploy a new > binary to the target and debug it, without waiting for a full image build. > > One approach to this is to build the component you want to debug. Then > directly copy the executable to BOTH the target and the host ‘debugfs’. If > the binary is processed through the debug splitting in OE, you should also > copy the debug items (the .debug contents and corresponding /usr/src/debug) > from the work dir. > > Such as: > > $ bitbake bash > $ bitbake -c devshell bash > $ cd .. > $ scp packages-split/bash/bin/bash <target>:/bin/bash > $ cp -a packages-split/bash-dbg/* <path>/debugfs You can see the new section at http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#platdev-gdb-remotedebug. It needs to be reviewed as I did some rewriting and guessing in some spots. Thanks, Scott 69577 15 srifenbark 2017-01-06 15:26:40 +0000 It has been quite some time since this change was made for the docs. I have yet to get an "okay" on the solution. Thanks, Scott 69763 16 mark.hatle 2017-01-12 16:36:50 +0000 Sorry thought I had responded previously. I've reviewed the new section and it appears to be correct and functional to me. 69916 17 srifenbark 2017-01-18 23:35:47 +0000 Mark, Thanks for getting back to me. I am marking the bug RESOLVED and setting the Doc flag to "done." Scott