While running the tools on target tests an issue was encountered when executing "perf test" command: root@qemuppc:~# perf test 1: vmlinux symtab matches kallsyms : FAILED! 2: detect open syscall event : FAILED! 3: detect open syscall event on all cpus : FAILED! 4: read samples using the mmap interface : FAILED! 5: parse events tests :Can't open event dir: No such file or directory Can't open event dir: No such file or directory Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 Warning: bad op token { Warning: Error: expected type 4 but read 0 FAILED! 6: Validate PERF_RECORD_* events & perf_sample fields : FAILED! 7: Test perf pmu format parsing : Ok 8: Test dso data interface : Ok 9: roundtrip evsel->name check : Ok 10: Check parsing of sched tracepoints fields : Ok 11: Generate and check syscalls:sys_enter_open event fields: FAILED! 12: struct perf_event_attr setup :sh: /media/sdd1/fb/casc/YoctoTest2/poky/poky-3e528d3e4bc542ca2f654c54363c56af0fb9f61b/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/python: No such file or directory FAILED! 13: Test matching and linking multiple hists : Ok 14: Try 'use perf' in python, checking link problems : FAILED! 15: Test breakpoint overflow signal handler : FAILED! 16: Test breakpoint overflow sampling : FAILED! 17: Test number of exit event of a simple workload : Skip 18: Test software clock events have valid period values : Ok 19: Test object code reading :*** Error in `perf': corrupted double-linked list: 0x1030ca60 *** ======= Backtrace: ========= /lib/libc.so.6[0xfe96f80] /lib/libc.so.6[0xfea3d28] /lib/libc.so.6[0xfea4e50] perf(machines__exit+0x94)[0x1007bd14] perf[0x100c12c4] perf(test__code_reading+0x28)[0x100c1d28] perf(cmd_test+0x39c)[0x1003b7bc] perf[0x10011870] perf(main+0x5e4)[0x10010da4] /lib/libc.so.6[0xfe4328c] /lib/libc.so.6[0xfe43428] ======= Memory map: ======== 00100000-00104000 r-xp 00000000 00:00 0 [vdso] 0f463000-0f464000 r-xp 00000000 03:00 22544 /usr/lib/traceevent/plugins/plugin_hrtimer.so 0f464000-0f473000 ---p 00001000 03:00 22544 /usr/lib/traceevent/plugins/plugin_hrtimer.so 0f473000-0f474000 rwxp 00000000 03:00 22544 /usr/lib/traceevent/plugins/plugin_hrtimer.so 0f484000-0f485000 r-xp 00000000 03:00 22552 /usr/lib/traceevent/plugins/plugin_kmem.so 0f485000-0f494000 ---p 00001000 03:00 22552 /usr/lib/traceevent/plugins/plugin_kmem.so 0f494000-0f495000 rwxp 00000000 03:00 22552 /usr/lib/traceevent/plugins/plugin_kmem.so 0f4a5000-0f4a6000 r-xp 00000000 03:00 22547 /usr/lib/traceevent/plugins/plugin_sched_switch.so 0f4a6000-0f4b5000 ---p 00001000 03:00 22547 /usr/lib/traceevent/plugins/plugin_sched_switch.so 0f4b5000-0f4b6000 rwxp 00000000 03:00 22547 /usr/lib/traceevent/plugins/plugin_sched_switch.so 0f4c6000-0f4c7000 r-xp 00000000 03:00 22551 /usr/lib/traceevent/plugins/plugin_mac80211.so 0f4c7000-0f4d6000 ---p 00001000 03:00 22551 /usr/lib/traceevent/plugins/plugin_mac80211.so 0f4d6000-0f4d7000 rwxp 00000000 03:00 22551 /usr/lib/traceevent/plugins/plugin_mac80211.so 0f4e7000-0f4e8000 r-xp 00000000 03:00 22548 /usr/lib/traceevent/plugins/plugin_function.so 0f4e8000-0f4f7000 ---p 00001000 03:00 22548 /usr/lib/traceevent/plugins/plugin_function.so 0f4f7000-0f4f8000 rwxp 00000000 03:00 22548 /usr/lib/traceevent/plugins/plugin_function.so 0f508000-0f50a000 r-xp 00000000 03:00 22545 /usr/lib/traceevent/plugins/plugin_scsi.so 0f50a000-0f519000 ---p 00002000 03:00 22545 /usr/lib/traceevent/plugins/plugin_scsi.so 0f519000-0f51a000 rwxp 00001000 03:00 22545 /usr/lib/traceevent/plugins/plugin_scsi.so 0f52a000-0f52b000 r-xp 00000000 03:00 22553 /usr/lib/traceevent/plugins/plugin_cfg80211.so 0f52b000-0f53a000 ---p 00001000 03:00 22553 /usr/lib/traceevent/plugins/plugin_cfg80211.so 0f53a000-0f53b000 rwxp 00000000 03:00 22553 /usr/lib/traceevent/plugins/plugin_cfg80211.so 0f54b000-0f54e000 r-xp 00000000 03:00 22549 /usr/lib/traceevent/plugins/plugin_kvm.so 0f54e000-0f55d000 ---p 00003000 03:00 22549 /usr/lib/traceevent/plugins/plugin_kvm.so 0f55d000-0f55e000 rwxp 00002000 03:00 22549 /usr/lib/traceevent/plugins/plugin_kvm.so 0f56e000-0f56f000 r-xp 00000000 03:00 22550 /usr/lib/traceevent/plugins/plugin_jbd2.so 0f56f000-0f57e000 ---p 00001000 03:00 22550 /usr/lib/traceevent/plugins/plugin_jbd2.so 0f57e000-0f57f000 rwxp 00000000 03:00 22550 /usr/lib/traceevent/plugins/plugin_jbd2.so 0f58f000-0f590000 r-xp 00000000 03:00 22546 /usr/lib/traceevent/plugins/plugin_xen.so 0f590000-0f59f000 ---p 00001000 03:00 22546 /usr/lib/traceevent/plugins/plugin_xen.so 0f59f000-0f5a0000 rwxp 00000000 03:00 22546 /usr/lib/traceevent/plugins/plugin_xen.so 0f5b0000-0f72b000 r-xp 00000000 03:00 22392 /usr/lib/libpython2.7.so.1.0 0f72b000-0f73b000 ---p 0017b000 03:00 22392 /usr/lib/libpython2.7.so.1.0 0f73b000-0f76e000 rwxp 0017b000 03:00 22392 /usr/lib/libpython2.7.so.1.0 0f76e000-0f777000 rwxp 00000000 00:00 0 0f780000-0f794000 r-xp 00000000 03:00 565 /lib/libz.so.1.2.8 0f794000-0f7a4000 ---p 00014000 03:00 565 /lib/libz.so.1.2.8 0f7a4000-0f7a5000 rwxp 00014000 03:00 565 /lib/libz.so.1.2.8 0f810000-0f8e8000 r-xp 00000000 03:00 21974 /usr/lib/libslang.so.2.2.4 0f8e8000-0f8f7000 ---p 000d8000 03:00 21974 /usr/lib/libslang.so.2.2.4 0f8f7000-0f90a000 rwxp 000d7000 03:00 21974 /usr/lib/libslang.so.2.2.4 0f90a000-0f948000 rwxp 00000000 00:00 0 0f950000-0faaa000 r-xp 00000000 03:00 23043 /usr/lib/libperl.so.5.14.3 0faaa000-0faba000 ---p 0015a000 03:00 23043 /usr/lib/libperl.so.5.14.3 0faba000-0fac4000 rwxp 0015a000 03:00 23043 /usr/lib/libperl.so.5.14.3 0fad0000-0fad8000 r-xp 00000000 03:00 342 /lib/libcrypt-2.19.so 0fad8000-0fae7000 ---p 00008000 03:00 342 /lib/libcrypt-2.19.so 0fae7000-0fae8000 r--p 00007000 03:00 342 /lib/libcrypt-2.19.so 0fae8000-0fae9000 rwxp 00008000 03:00 342 /lib/libcrypt-2.19.so 0fae9000-0fb10000 rwxp 00000000 00:00 0 0fb90000-0fba5000 r-xp 00000000 03:00 41381 /usr/lib/libelf-0.148.so 0fba5000-0fbb5000 ---p 00015000 03:00 41381 /usr/lib/libelf-0.148.so 0fbb5000-0fbb6000 r--p 00015000 03:00 41381 /usr/lib/libelf-0.148.so 0fbb6000-0fbb7000 rwxp 00016000 03:00 41381 /usr/lib/libelf-0.148.so 0fbc0000-0fcb1000 r-xp 00000000 03:00 21372 /usr/lib/libbfd-2.24.so 0fcb1000-0fcc1000 ---p 000f1000 03:00 21372 /usr/lib/libbfd-2.24.so 0fcc1000-0fcd0000 rwxp 000f1000 03:00 21372 /usr/lib/libbfd-2.24.so 0fcd0000-0fcd5000 rwxp 00000000 00:00 0 0fce0000-0fce2000 r-xp 00000000 03:00 370 /lib/libutil-2.19.so 0fce2000-0fcf1000 ---p 00002000 03:00 370 /lib/libutil-2.19.so 0fcf1000-0fcf2000 r--p 00001000 03:00 370 /lib/libutil-2.19.so 0fcf2000-0fcf3000 rwxp 00002000 03:00 370 /lib/libutil-2.19.so 0fd00000-0fd07000 r-xp 00000000 03:00 590 /lib/librt-2.19.so 0fd07000-0fd16000 ---p 00007000 03:00 590 /lib/librt-2.19.so 0fd16000-0fd17000 r--p 00006000 03:00 590 /lib/librt-2.19.so 0fd17000-0fd18000 rwxp 00007000 03:00 590 /lib/librt-2.19.so 0fd20000-0fdcc000 r-xp 00000000 03:00 360 /lib/libm-2.19.so 0fdcc000-0fddb000 ---p 000ac000 03:00 360 /lib/libm-2.19.so 0fddb000-0fddc000 r--p 000ab000 03:00 360 /lib/libm-2.19.so 0fddc000-0fde0000 rwxp 000ac000 03:00 360 /lib/libm-2.19.so 0fdf0000-0fe07000 r-xp 00000000 03:00 361 /lib/libpthread-2.19.so 0fe07000-0fe16000 ---p 00017000 03:00 361 /lib/libpthread-2.19.so 0fe16000-0fe17000 r--p 00016000 03:00 361 /lib/libpthread-2.19.so 0fe17000-0fe18000 rwxp 00017000 03:00 361 /lib/libpthread-2.19.so 0fe18000-0fe1a000 rwxp 00000000 00:00 0 0fe20000-0ff84000 r-xp 00000000 03:00 564 /lib/libc-2.19.so 0ff84000-0ff94000 ---p 00164000 03:00 564 /lib/libc-2.19.so 0ff94000-0ff96000 r--p 00164000 03:00 564 /lib/libc-2.19.so 0ff96000-0ff9a000 rwxp 00166000 03:00 564 /lib/libc-2.19.so 0ff9a000-0ff9c000 rwxp 00000000 00:00 0 0ffa0000-0ffa3000 r-xp 00000000 03:00 575 /lib/libdl-2.19.so 0ffa3000-0ffb2000 ---p 00003000 03:00 575 /lib/libdl-2.19.so 0ffb2000-0ffb3000 r--p 00002000 03:00 575 /lib/libdl-2.19.so 0ffb3000-0ffb4000 rwxp 00003000 03:00 575 /lib/libdl-2.19.so 0ffc0000-0ffe0000 r-xp 00000000 03:00 289 /lib/ld-2.19.so 0ffef000-0fff0000 r--p 0001f000 03:00 289 /lib/ld-2.19.so 0fff0000-0fff2000 rwxp 00020000 03:00 289 /lib/ld-2.19.so 10000000-10126000 r-xp 00000000 03:00 16008 /usr/bin/perf 10135000-1013d000 rwxp 00125000 03:00 16008 /usr/bin/perf 1013d000-10418000 rwxp 00000000 00:00 0 [heap] b7800000-b7821000 rw-p 00000000 00:00 0 b7821000-b7900000 ---p 00000000 00:00 0 b7960000-b7965000 rw-p 00000000 00:00 0 b796f000-b7970000 rw-p 00000000 00:00 0 bfb07000-bfb28000 rw-p 00000000 00:00 0 [stack] Aborted
Additional information: Test object: 1.6_M5_RC2:3e528d3e4bc542ca2f654c54363c56af0fb9f61b Test target: qemuPPC Test host: Ubuntu 12.04
Do you happen to know if "perf test" fails only on ppc? Did you happen to test on a qemux86 or x86_64 - does that work for you?
.
Tom, I see the same "bad op token" on x86_64. Is this something you have seen? Is there a usage error here maybe?
perf and other tracing tools should not be tested on emulated machines, only real hardware - they lack pmu emulation etc.
I'll confirm on real hardware if the problem goes away - thanks Tom.
Confirming Tom's statement, when run on real x86_64 hardware, I do not observe the following errors: Warning: Error: expected type 4 but read 0 Warning: bad op token { I do still see a few errors: 1: vmlinux symtab matches kallsyms : FAILED! 7: Validate PERF_RECORD_* events & perf_sample fields : FAILED! 13: struct perf_event_attr setup :sh: /build/yocto/master/intel-corei7-64_20140114211153/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/python: No such file or directory FAILED! 15: Try 'use perf' in python, checking link problems : FAILED! Tom, what are your thoughts on these? 1) We have /proc/kallsysm, not sure what symtab is 2) Does some sampling run need to be run manually first? 13) A sysroot issue? 15) Python is installed and runs a simple print test, are there some debug or other dependencies the image needs to install possibly?
(In reply to comment #8) > Confirming Tom's statement, when run on real x86_64 hardware, I do not > observe the following errors: > > Warning: Error: expected type 4 but read 0 > Warning: bad op token { > > I do still see a few errors: > > 1: vmlinux symtab matches kallsyms : FAILED! > 7: Validate PERF_RECORD_* events & perf_sample fields : FAILED! > 13: struct perf_event_attr setup :sh: > /build/yocto/master/intel-corei7-64_20140114211153/build/tmp/sysroots/x86_64- > linux/usr/bin/python-native/python: No such file or directory > FAILED! > 15: Try 'use perf' in python, checking link problems : FAILED! > > Tom, what are your thoughts on these? > We haven't used perf test as a functional test in the past, but should probably make sure it runs cleanly for 1.7. Despite the warnings, basic perf functionality works fine. In any case, we'll need to go through each failure and track down the cause, but verbose output currently shows the following: > 1) We have /proc/kallsysm, not sure what symtab is Looking at the vmlinux_path (6 entries long) symsrc__init: cannot get elf header. Using /boot/vmlinux-3.10.32-ltsi-yocto-standard for symbols Maps only in vmlinux: c1971000-c1a01057 0 [kernel].init.text c1a01058-ffffffffffffffff 0 [kernel].exit.text > 2) Does some sampling run need to be run manually first? No, it looks like a consequence of not using real hardware. With real hardware: 2: detect open syscall event : --- start --- ---- end ---- detect open syscall event: Ok > 13) A sysroot issue? Probably a missing native dependency. > 15) Python is installed and runs a simple print test, are there some debug > or other dependencies the image needs to install possibly? Probably same cause as 13.
Moving to 1.7. And note that qemu is explicitly *not* included.
(In reply to comment #10) > Moving to 1.7. And note that qemu is explicitly *not* included. Hi Tom, Darren, Do you need any more investigation from QA on this?
Ee Peng, can you please check if this is still an issue and close this if it isn't? I can't reproduce this now on latest.
The code has evolved since the time this was reported and I can no longer reproduce this.
Ptest perf command works fine with 2.4.4rc1 and 2.6M4RC1. While running "perf test" command works fine with 2.4.4rc1 on NUC6, It works fine, Please find the logs for the same : root@genericx86-64:~# perf test 1: vmlinux symtab matches kallsyms : Skip 2: Detect openat syscall event : Ok 3: Detect openat syscall event on all cpus : Ok 4: Read samples using the mmap interface : Ok 5: Parse event definition strings : Ok 6: Simple expression parser : Ok 7: PERF_RECORD_* events & perf_sample fields : FAILED! 8: Parse perf pmu format : Ok 9: DSO data read : Ok 10: DSO data cache : Ok 11: DSO data reopen : Ok 12: Roundtrip evsel->name : Ok 13: Parse sched tracepoints fields : Ok 14: syscalls:sys_enter_openat event fields : Ok 15: Setup struct perf_event_attr : Skip 16: Match and link multiple hists : Ok 17: 'import perf' in python : FAILED! 18: Breakpoint overflow signal handler : Ok 19: Breakpoint overflow sampling : Ok 20: Number of exit events of a simple workload : Ok 21: Software clock events period values : Ok 22: Object code reading : Ok 23: Sample parsing : Ok 24: Use a dummy software event to keep tracking: Ok 25: Parse with no sample_id_all bit set : Ok 26: Filter hist entries : Ok 27: Lookup mmap thread : Ok 28: Share thread mg : Ok 29: Sort output of hist entries : Ok 30: Cumulate child hist entries : Ok 31: Track with sched_switch : Ok 32: Filter fds with revents mask in a fdarray : Ok 33: Add fd to a fdarray, making it autogrow : Ok 34: kmod_path__parse : Ok 35: Thread map : Ok 36: LLVM search and compile : 36.1: Basic BPF llvm compile : Skip 36.2: kbuild searching : Skip 36.3: Compile source for BPF prologue generation: Skip 36.4: Compile source for BPF relocation : Skip 37: Session topology : Ok 38: BPF filter : 38.1: Basic BPF filtering : Skip 38.2: BPF pinning : Skip 38.3: BPF relocation checker : Skip 39: Synthesize thread map : Ok 40: Remove thread map : Ok 41: Synthesize cpu map : Ok 42: Synthesize stat config : Ok 43: Synthesize stat : Ok 44: Synthesize stat round : Ok 45: Synthesize attr update : Ok 46: Event times : Ok 47: Read backward ring buffer : Ok 48: Print cpu map : Ok 49: Probe SDT events : Skip 50: is_printable_array : Ok 51: Print bitmap : Ok 52: perf hooks : Ok 53: builtin clang support : Skip (not compiled in) 54: unit_number__scnprintf : Ok 55: x86 rdpmc : Ok 56: Convert perf time to TSC : Ok 57: DWARF unwind : FAILED! 58: x86 instruction decoder - new instructions : Ok While running "perf test" command works fine with 2.6M4_rc1 on NUC6, It works fine, Please find the logs for the same: root@genericx86-64:~# perf test 1: vmlinux symtab matches kallsyms : Skip 2: Detect openat syscall event : Ok 3: Detect openat syscall event on all cpus : Ok 4: Read samples using the mmap interface : Ok 5: Test data source output : Ok 6: Parse event definition strings : Ok 7: Simple expression parser : Ok 8: PERF_RECORD_* events & perf_sample fields : FAILED! 9: Parse perf pmu format : Ok 10: DSO data read : Ok 11: DSO data cache : Ok 12: DSO data reopen : Ok 13: Roundtrip evsel->name : Ok 14: Parse sched tracepoints fields : Ok 15: syscalls:sys_enter_openat event fields : Ok 16: Setup struct perf_event_attr : Skip 17: Match and link multiple hists : Ok 18: 'import perf' in python : FAILED! 19: Breakpoint overflow signal handler : Ok 20: Breakpoint overflow sampling : Ok 21: Breakpoint accounting : Ok 22: Number of exit events of a simple workload : Ok 23: Software clock events period values : Ok 24: Object code reading : Ok 25: Sample parsing : Ok 26: Use a dummy software event to keep tracking: Ok 27: Parse with no sample_id_all bit set : Ok 28: Filter hist entries : Ok 29: Lookup mmap thread : Ok 30: Share thread mg : Ok 31: Sort output of hist entries : Ok 32: Cumulate child hist entries : Ok 33: Track with sched_switch : Ok 34: Filter fds with revents mask in a fdarray : Ok 35: Add fd to a fdarray, making it autogrow : Ok 36: kmod_path__parse : Ok 37: Thread map : Ok 38: LLVM search and compile : 38.1: Basic BPF llvm compile : Skip 38.2: kbuild searching : Skip 38.3: Compile source for BPF prologue generation: Skip 38.4: Compile source for BPF relocation : Skip 39: Session topology : Ok 40: BPF filter : 40.1: Basic BPF filtering : FAILED! 40.2: BPF pinning : Skip 40.3: BPF relocation checker : Skip 41: Synthesize thread map : Ok 42: Remove thread map : Ok 43: Synthesize cpu map : Ok 44: Synthesize stat config : Ok 45: Synthesize stat : Ok 46: Synthesize stat round : Ok 47: Synthesize attr update : Ok 48: Event times : Ok 49: Read backward ring buffer : Ok 50: Print cpu map : Ok 51: Probe SDT events : Skip 52: is_printable_array : Ok 53: Print bitmap : Ok 54: perf hooks : Ok 55: builtin clang support : Skip (not compiled in) 56: unit_number__scnprintf : Ok 57: mem2node : Ok 58: x86 rdpmc : Ok 59: Convert perf time to TSC : Ok 60: DWARF unwind : FAILED! 61: x86 instruction decoder - new instructions : Ok