3756 2013-01-18 13:22:04 +0000 crosstap and stap fail in core-image-sato-sdk (pre 1.4M3) 2013-02-26 10:08:06 +0000 1 1 7 Build System, Metadata & Runtime Meta-yocto meta-yocto 1.4 x86 Multiple RESOLVED FIXED Medium normal 1.4 M4 1 sstncr tom.zanussi poky.bs.watcher poky.watcher sgw --- --- --- oldest_to_newest 28940 0 sstncr 2013-01-18 13:22:04 +0000 commit 9eb88ceb39b7d0b8ddc6487e61ce8edadef10ec4 1.Create the trace_open.stp script as follows: probe syscall.open { printf ("%s(%d) open (%s)\n", execname(), pid(), argstr) } 2.Bitbake a core-image-sato-sdk image and start it under qemu. 3.Make sure that the ssh service is started on the Qemu machine. 4.From the host machine poky build_dir, run "crosstap root@192.168.7.2 trace_open.stp". The script should successfully connect to the qemu machine and there should be presented a list of services(pid, process name) which run on the qemu machine. Output: [stefans@firebird yotta]$ crosstap root@192.168.7.2 trace_open.stp ~/yocto/builds/yotta ~/yocto/builds/yotta ~/yocto/builds/yotta Checking "/lib/modules/3.4.24-yocto-standard/build/.config" failed with error: No such file or directory [stefans@firebird yotta]$ Same thing if I run stap directly in the image: root@qemux86-64:~# stap trace_open.stp Checking "/lib/modules/3.4.24-yocto-standard/build/.config" failed with error: No such file or directory root@qemux86-64:~# 29204 1 sstncr 2013-01-25 16:33:59 +0000 There is a change in commit 8d6e55bf2192524bda84138b5356a2791adbbe61 The output changed a little bit, but the error is the same (again this is a core-image-sato-sdk) [stefans@firebird yotta]$ crosstap root@192.168.7.2 trace_open.stp ~/yocto/builds/yotta ~/yocto/builds/yotta ~/yocto/builds/yotta Error: No target kernel build found. Did you forget to create a local build of your image? 'crosstap' requires a local sdk build of the target system (or a build that includes 'tools-profile') in order to build kernel modules that can probe the target system. Practically speaking, that means you need to do the following: - If you're running a pre-built image, download the release and/or BSP tarballs used to build the image. - If you're working from git sources, just clone the metadata and BSP layers needed to build the image you'll be booting. - Make sure you're properly set up to build a new image (see the BSP README and/or the widely available basic documentation that discusses how to build images). - Build an -sdk version of the image e.g.: $ bitbake core-image-sato-sdk OR - Build a non-sdk image but include the profiling tools: [ edit local.conf and add 'tools-profile' to the end of the EXTRA_IMAGE_FEATURES variable ] $ bitbake core-image-sato [ NOTE that 'crosstap' needs to be able to ssh into the target system, which isn't enabled by default in -minimal images. ] Once you've build the image on the host system, you're ready to boot it (or the equivalent pre-built image) and use 'crosstap' to probe it (you need to source the environment as usual first): $ source oe-init-build-env $ cd ~/my/systemtap/scripts $ crosstap root@192.168.1.xxx myscript.stp [stefans@firebird yotta]$ But on the image: [stefans@firebird yotta]$ scp trace_open.stp root@192.168.7.2:~/ trace_open.stp 100% 89 0.1KB/s 00:00 [stefans@firebird yotta]$ ssh root@192.168.7.2 root@qemux86-64:~# cat trace_open.stp probe syscall.open { printf ("%s(%d) open (%s)\n", execname(), pid(), argstr) } root@qemux86-64:~# stap trace_open.stp Checking "/lib/modules/3.4.26-yocto-standard/build/.config" failed with error: No such file or directory root@qemux86-64:~# root@qemux86-64:~# rpm -qa | grep systemtap systemtap-1.8+git0+48fa6b5e1d3e636c34cbd5650370e6f634efc294-r0.x86_64 systemtap-locale-en-1.8+git0+48fa6b5e1d3e636c34cbd5650370e6f634efc294-r0.x86_64 systemtap-dev-1.8+git0+48fa6b5e1d3e636c34cbd5650370e6f634efc294-r0.x86_64 root@qemux86-64:~# 29265 2 tom.zanussi 2013-01-29 00:35:21 +0000 I've verified that the path to the build directory containing .config is correct and is being passed into stap. Something is causing stap to ignore the passed-in value, investigating. 29426 3 tom.zanussi 2013-01-31 23:23:21 +0000 Digging around in the systemtap code, the ssh_remote class's connect() method clones the associated sytemtap_session, but the copy constructor for systemtap_session doesn't copy the .kernel_build_tree member, which was set by the -r command-line option. Copying kernel_build_tree in the constructor fixes the problem, but I need to do it in the correct 'C++ way'. 29477 4 tom.zanussi 2013-02-02 00:39:45 +0000 Submitted to oe-core list and upstream to the systemtap list. 29508 5 tom.zanussi 2013-02-05 01:26:05 +0000 Turns out the problem is passing in x86-64 instead of the underscore version to the -a param of stap. Testing a fix to the crosstap script now. 29587 6 tom.zanussi 2013-02-06 16:41:50 +0000 Fixed in master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=97b7e4f46e2116e9cbaa02efa7d37321fd32b368 30070 7 sstncr 2013-02-25 10:58:56 +0000 commmit: 7d22ef28e64de06b6238d0ef9b664c0bee6efa46 I'm seeing the same behaviour with stap/crosstap: root@qemux86-64:~# cat trace_open.stp probe syscall.open { printf ("%s(%d) open (%s)\n", execname(), pid(), argstr) } root@qemux86-64:~# stap trace_open.stp Checking "/lib/modules/3.4.28-yocto-standard/build/.config" failed with error: No such file or directory [stefans@firebird yotta]$ grep 64 `which crosstap` x86?64*) SYSTEMTAP_TARGET_ARCH="x86_64" [stefans@firebird yotta]$ ssh root@192.168.7.2 'which stap' /usr/bin/stap [stefans@firebird yotta]$ crosstap root@192.168.7.2 trace_open.stp ~/yocto/builds/yotta ~/yocto/builds/yotta ~/yocto/builds/yotta Error: No target kernel build found. Did you forget to create a local build of your image? 'crosstap' requires a local sdk build of the target system (or a build that includes 'tools-profile') in order to build kernel modules that can probe the target system. Practically speaking, that means you need to do the following: - If you're running a pre-built image, download the release and/or BSP tarballs used to build the image. - If you're working from git sources, just clone the metadata and BSP layers needed to build the image you'll be booting. - Make sure you're properly set up to build a new image (see the BSP README and/or the widely available basic documentation that discusses how to build images). - Build an -sdk version of the image e.g.: $ bitbake core-image-sato-sdk OR - Build a non-sdk image but include the profiling tools: [ edit local.conf and add 'tools-profile' to the end of the EXTRA_IMAGE_FEATURES variable ] $ bitbake core-image-sato [ NOTE that 'crosstap' needs to be able to ssh into the target system, which isn't enabled by default in -minimal images. ] Once you've build the image on the host system, you're ready to boot it (or the equivalent pre-built image) and use 'crosstap' to probe it (you need to source the environment as usual first): $ source oe-init-build-env $ cd ~/my/systemtap/scripts $ crosstap root@192.168.1.xxx myscript.stp [stefans@firebird yotta]$ 30093 8 tom.zanussi 2013-02-26 00:35:54 +0000 (In reply to comment #7) > commmit: 7d22ef28e64de06b6238d0ef9b664c0bee6efa46 > > I'm seeing the same behaviour with stap/crosstap: > root@qemux86-64:~# cat trace_open.stp > probe syscall.open > { > printf ("%s(%d) open (%s)\n", execname(), pid(), argstr) > } > root@qemux86-64:~# stap trace_open.stp > Checking "/lib/modules/3.4.28-yocto-standard/build/.config" failed with > error: No such file or directory > > [stefans@firebird yotta]$ grep 64 `which crosstap` > x86?64*) > SYSTEMTAP_TARGET_ARCH="x86_64" > > [stefans@firebird yotta]$ ssh root@192.168.7.2 'which stap' > /usr/bin/stap > [stefans@firebird yotta]$ crosstap root@192.168.7.2 trace_open.stp > ~/yocto/builds/yotta ~/yocto/builds/yotta > ~/yocto/builds/yotta > > Error: No target kernel build found. > Did you forget to create a local build of your image? > > 'crosstap' requires a local sdk build of the target system > (or a build that includes 'tools-profile') in order to build > kernel modules that can probe the target system. > > Practically speaking, that means you need to do the following: > - If you're running a pre-built image, download the release > and/or BSP tarballs used to build the image. > - If you're working from git sources, just clone the metadata > and BSP layers needed to build the image you'll be booting. > - Make sure you're properly set up to build a new image (see > the BSP README and/or the widely available basic documentation > that discusses how to build images). > - Build an -sdk version of the image e.g.: > $ bitbake core-image-sato-sdk > OR > - Build a non-sdk image but include the profiling tools: > [ edit local.conf and add 'tools-profile' to the end of > the EXTRA_IMAGE_FEATURES variable ] > $ bitbake core-image-sato > > [ NOTE that 'crosstap' needs to be able to ssh into the target > system, which isn't enabled by default in -minimal images. ] > > Once you've build the image on the host system, you're ready to > boot it (or the equivalent pre-built image) and use 'crosstap' > to probe it (you need to source the environment as usual first): > > $ source oe-init-build-env > $ cd ~/my/systemtap/scripts > $ crosstap root@192.168.1.xxx myscript.stp > > [stefans@firebird yotta]$ It looks like this time it's not finding the .config because the kernel hasn't been built on the host system, which is required as noted in the instructions. Have you done what the message says to do for the prerequisites? Here's a transcript of running trace_open.stp that worked out of the box for me: [trz@empanada crosstap-test]$ git checkout -b commit7d22ef28e64de06b6238d0ef9b664c0bee6efa46 7d22ef28e64de06b6\ 238d0ef9b664c0bee6efa46 [trz@empanada crosstap-test]$ source oe-init-build-env [trz@empanada build]$ bitbake core-image-sato-sdk [trz@empanada build]$ runqemu qemux86-64 ^Z [1]+ Stopped runqemu qemux86-64 [trz@empanada build]$ bg 1 [trz@empanada build]$ cd /home/trz/tracing/systemtap/scripts/tutorial [trz@empanada tutorial]$ crosstap root@192.168.7.2 trace_open.stp ~/tracing/systemtap/scripts/tutorial ~/tracing/systemtap/scripts/tutorial ~/tracing/systemtap/scripts/tutorial The authenticity of host '192.168.7.2 (192.168.7.2)' can't be established. RSA key fingerprint is 67:bd:49:90:55:63:bf:5b:5d:73:36:3d:94:ae:33:77. Are you sure you want to continue connecting (yes/no)? yes ~/tracing/systemtap/scripts/tutorial ls(704) open ("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/tls/x86_64/libcap.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/tls/libcap.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/x86_64/libcap.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libcap.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libacl.so.1", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libc.so.6", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libattr.so.1", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libnss_compat.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libnsl.so.1", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libnss_nis.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/etc/passwd", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/etc/group", O_RDONLY|O_CLOEXEC|O_CLOEXEC) ls(704) open ("/usr/share/zoneinfo/UTC", O_RDONLY|O_CLOEXEC|O_CLOEXEC) matchbox-termin(663) open ("/tmp/vte1S62SW", O_RDWR|O_CREAT|O_EXCL, 0600) matchbox-termin(663) open ("/tmp/vteK35RSW", O_RDWR|O_CREAT|O_EXCL, 0600) matchbox-termin(663) open ("/tmp/vte9V6RSW", O_RDWR|O_CREAT|O_EXCL, 0600) 30106 9 sstncr 2013-02-26 10:08:06 +0000 (In reply to comment #8) > > It looks like this time it's not finding the .config because the kernel > hasn't been built on the host system, which is required as noted in the > instructions. Have you done what the message says to do for the > prerequisites? > Something is wierd because I did had a build of core-image-sato-sdk, as the instructions recommend. However it was likely rebuilt mostly from sstate. So I wiped my tmp dir, did a bitbake -ccleansstate linux-yocto, and rebuilt core-image-sato-sdk and now indeed it works as advertised.