Bug 15201

Summary: perf do_compile fails: "libtraceevent is missing"
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Luca Ceresoli <luca.ceresoli>
Component: coreAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Undecided CC: alexandre.belloni, jon.mason, luca.ceresoli, meta.mr.watcher, meta.watcher, ross.burton
Version: unspecified   
Target Milestone: 0.0.0   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Luca Ceresoli 2023-08-26 08:08:58 UTC
Seen on the autobuilders, also reported by Ross Burton and Alexandre Belloni:

oe_runmake failed
ERROR: perf-1.0-r9 do_compile: ExecutionError('/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761', 1, None, None)
NOTE: recipe groff-1.22.4-r1: task do_compile: Succeeded
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/log.do_compile.1455761
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: make -j 16 -l 52 V=1 VF=1 -C /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/perf O=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0 CROSS_COMPILE=mips64-poky-linux- ARCH=mips CC=mips64-poky-linux-gcc  -meb -mabi=64 -mhard-float -march=mips64r2 -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot=  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot=  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot-native=  --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot CCLD=mips64-poky-linux-gcc  -meb -mabi=64 -mhard-float -march=mips64r2 -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot=  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot=  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot-native=  --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot LDSHARED=mips64-poky-linux-gcc  -meb -mabi=64 -mhard-float -march=mips64r2 -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9  -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot=  -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot=  -fdebug-prefix-map=/home/pokybuild/yocto-w
rker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot-native=  --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot -shared AR=mips64-poky-linux-gcc-ar LD=mips64-poky-linux-ld --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot  -m elf64btsmip EXTRA_CFLAGS=-ldw -I/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0 YFLAGS=-y --file-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0=/usr/src/debug/perf/1.0-r9 EXTRA_LDFLAGS=-m elf64btsmip perfexecdir=/usr/libexec NO_GTK2=1 NO_LIBAUDIT=1 NO_LIBBABELTRACE=1 NO_LIBBFD=1 NO_DWARF=1 NO_JVMTI=1 NO_LIBNUMA=1 NO_LIBPERL=1 NO_LIBPFM4=1 NO_SDT=1 PKG_CONFIG=pkg-config TMPDIR=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0 LIBUNWIND_DIR=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot/usr DESTDIR=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/image prefix=/usr bindir=/usr/bin sharedir=/usr/share sysconfdir=/etc perfexecdir=/usr/libexec/perf-core ETC_PERFCONFIG=../etc sharedir=share mandir=share/man infodir=share/info PYTHON=python3 PYTHON_CONFIG=python3-config all
| make: Entering directory '/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/perf'
|   BUILD:   Doing 'make -j48' parallel build
| make -C /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/build CFLAGS= LDFLAGS= /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/fixdep
| make -f /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/build/Makefile.build dir=. obj=fixdep
| gcc  -o /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/fixdep /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/fixdep-in.o
| make FIXDEP=1 -f Makefile.perf
| Makefile.config:594: DWARF support is off, BPF prologue is disabled
| Makefile.config:1170: *** ERROR: libtraceevent is missing. Please install libtraceevent-dev/libtraceevent-devel or build with NO_LIBTRACEEVENT=1.  Stop.
| make[1]: *** [Makefile.perf:235: sub-make] Error 2
| ERROR: oe_runmake failed
| make: Leaving directory '/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/perf'
| WARNING: /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761:194 exit 1 from 'exit 1'
| WARNING: Backtrace (BB generated script):
| make: *** [Makefile:70: all] Error 2
| 	#1: bbfatal_log, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 194
| 	#2: die, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 178
| 	#3: oe_runmake, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 166
| 	#4: do_compile, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 161
| 	#5: main, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 207
NOTE: recipe perf-1.0-r9: task do_compile: Failed
ERROR: Task (/home/pokybuild/yocto-worker/qemumips64/build/meta/recipes-kernel/perf/perf.bb:do_compile) failed with exit code '1'

https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/7609/steps/11/logs/stdio

qemumips64 on fedora38-ty-1

https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/7611/steps/11/logs/stdio

qemumips64 on fedora37-ty-2
Comment 1 Luca Ceresoli 2023-08-26 08:14:29 UTC
Saved builddir on fedora38-ty-1 to ~pokybuild/build-qemumips64-7609-perf-compile-failure

Pushed tag 'lucaceresoli/bug-15201-perf-libtraceevent-missing' on the contrib oe-core and bitbake repositories with the commits that triggered the failures.
Comment 2 Ross Burton 2023-08-26 12:25:25 UTC
FWIW, meta-arm CI has been seeing this in _some_ builds for a week now.  Our CI hits it reliably for some machines but I can't replicate using the identical configuration on my builder, so we've a hunch that something bad has got into sstate.
Comment 3 Ross Burton 2023-08-26 12:48:04 UTC
The underlying issue is that the libtraceevent feature check (which basically just links a stub piece of code with -ltraceevent) fails:

ld: libtraceevent.so: undefined reference to `kbuffer_alloc'

This is defined in src/kbuffer-parse.c in libtraceevent which should have been linked into the library but:

$ nm -D ./image/usr/lib/libtraceevent.so.1.7.3|grep kbuffer
                 U kbuffer_alloc

It's undefined.

On my local build of libtraceevent:

$ nm -D libtraceevent.so.1.7.3|grep kbuffer
0000000000013da0 T kbuffer_alloc

So as Jon suspected on Thursday, it's a libtraceevent issue. Once the bad build for libtraceevent gets into sstate, perf will always fail.
Comment 4 Luca Ceresoli 2023-08-27 10:10:25 UTC
As Ross predicted, it is now happening all the time:

https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/7618/steps/11/logs/stdio

qemumips64 on rocky9-ty-1
Comment 5 Ross Burton 2023-08-27 19:22:55 UTC
$ make -j libs
  CLEAN              plugins
  CLEAN              src
  CLEAN              libtraceevent
  DESCEND            src libtraceevent.a
  DESCEND            src libtraceevent.so
  COMPILE FPIC       kbuffer-parse.o
  COMPILE FPIC       parse-filter.o
  COMPILE FPIC       parse-utils.o
  COMPILE FPIC       event-parse-api.o
  COMPILE FPIC       event-parse-api.o
  COMPILE FPIC       kbuffer-parse.o
  COMPILE FPIC       tep_strerror.o
  COMPILE FPIC       tep_strerror.o
  COMPILE FPIC       parse-utils.o
  COMPILE FPIC       trace-seq.o
  COMPILE FPIC       parse-filter.o
  COMPILE FPIC       trace-seq.o
  COMPILE FPIC       event-plugin.o
  COMPILE FPIC       event-plugin.o
  COMPILE FPIC       event-parse.o
  COMPILE FPIC       event-parse.o
  COMPILE SHARED LIB libtraceevent.so.1.7.3
  BUILD STATIC LIB   libtraceevent.a

Note that it compiles kbuffer-parse.o _twice_, so one of the library links is probably reading the .o whilst it's being rewritten.

I'll switch the recipe to use Meson.
Comment 6 Ross Burton 2023-08-27 19:57:57 UTC
Fix sent, "[PATCH] libtraceevent: build with Meson".
Comment 7 Luca Ceresoli 2023-08-28 16:01:51 UTC
The build I had run and that had been consistently failing has succeeded after applying Ross's patch:

https://lore.kernel.org/openembedded-core/20230827195744.3451852-1-ross.burton@arm.com/T/#u
Comment 8 Ross Burton 2023-08-29 17:32:26 UTC
Merged in 5d060725e706476751b0bda8e6d9f3670aa4b7de.