Seen on the autobuilders, also reported by Ross Burton and Alexandre Belloni: oe_runmake failed ERROR: perf-1.0-r9 do_compile: ExecutionError('/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761', 1, None, None) NOTE: recipe groff-1.22.4-r1: task do_compile: Succeeded ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/log.do_compile.1455761 Log data follows: | DEBUG: Executing shell function do_compile | NOTE: make -j 16 -l 52 V=1 VF=1 -C /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/perf O=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0 CROSS_COMPILE=mips64-poky-linux- ARCH=mips CC=mips64-poky-linux-gcc -meb -mabi=64 -mhard-float -march=mips64r2 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot= -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot= -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot-native= --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot CCLD=mips64-poky-linux-gcc -meb -mabi=64 -mhard-float -march=mips64r2 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot= -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot= -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot-native= --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot LDSHARED=mips64-poky-linux-gcc -meb -mabi=64 -mhard-float -march=mips64r2 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot= -fmacro-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot= -fdebug-prefix-map=/home/pokybuild/yocto-w rker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot-native= --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot -shared AR=mips64-poky-linux-gcc-ar LD=mips64-poky-linux-ld --sysroot=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot -m elf64btsmip EXTRA_CFLAGS=-ldw -I/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0 YFLAGS=-y --file-prefix-map=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0=/usr/src/debug/perf/1.0-r9 EXTRA_LDFLAGS=-m elf64btsmip perfexecdir=/usr/libexec NO_GTK2=1 NO_LIBAUDIT=1 NO_LIBBABELTRACE=1 NO_LIBBFD=1 NO_DWARF=1 NO_JVMTI=1 NO_LIBNUMA=1 NO_LIBPERL=1 NO_LIBPFM4=1 NO_SDT=1 PKG_CONFIG=pkg-config TMPDIR=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0 LIBUNWIND_DIR=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/recipe-sysroot/usr DESTDIR=/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/image prefix=/usr bindir=/usr/bin sharedir=/usr/share sysconfdir=/etc perfexecdir=/usr/libexec/perf-core ETC_PERFCONFIG=../etc sharedir=share mandir=share/man infodir=share/info PYTHON=python3 PYTHON_CONFIG=python3-config all | make: Entering directory '/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/perf' | BUILD: Doing 'make -j48' parallel build | make -C /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/build CFLAGS= LDFLAGS= /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/fixdep | make -f /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/build/Makefile.build dir=. obj=fixdep | gcc -o /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/fixdep /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/fixdep-in.o | make FIXDEP=1 -f Makefile.perf | Makefile.config:594: DWARF support is off, BPF prologue is disabled | Makefile.config:1170: *** ERROR: libtraceevent is missing. Please install libtraceevent-dev/libtraceevent-devel or build with NO_LIBTRACEEVENT=1. Stop. | make[1]: *** [Makefile.perf:235: sub-make] Error 2 | ERROR: oe_runmake failed | make: Leaving directory '/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/perf-1.0/tools/perf' | WARNING: /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761:194 exit 1 from 'exit 1' | WARNING: Backtrace (BB generated script): | make: *** [Makefile:70: all] Error 2 | #1: bbfatal_log, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 194 | #2: die, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 178 | #3: oe_runmake, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 166 | #4: do_compile, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 161 | #5: main, /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/perf/1.0/temp/run.do_compile.1455761, line 207 NOTE: recipe perf-1.0-r9: task do_compile: Failed ERROR: Task (/home/pokybuild/yocto-worker/qemumips64/build/meta/recipes-kernel/perf/perf.bb:do_compile) failed with exit code '1' https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/7609/steps/11/logs/stdio qemumips64 on fedora38-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/7611/steps/11/logs/stdio qemumips64 on fedora37-ty-2
Saved builddir on fedora38-ty-1 to ~pokybuild/build-qemumips64-7609-perf-compile-failure Pushed tag 'lucaceresoli/bug-15201-perf-libtraceevent-missing' on the contrib oe-core and bitbake repositories with the commits that triggered the failures.
FWIW, meta-arm CI has been seeing this in _some_ builds for a week now. Our CI hits it reliably for some machines but I can't replicate using the identical configuration on my builder, so we've a hunch that something bad has got into sstate.
The underlying issue is that the libtraceevent feature check (which basically just links a stub piece of code with -ltraceevent) fails: ld: libtraceevent.so: undefined reference to `kbuffer_alloc' This is defined in src/kbuffer-parse.c in libtraceevent which should have been linked into the library but: $ nm -D ./image/usr/lib/libtraceevent.so.1.7.3|grep kbuffer U kbuffer_alloc It's undefined. On my local build of libtraceevent: $ nm -D libtraceevent.so.1.7.3|grep kbuffer 0000000000013da0 T kbuffer_alloc So as Jon suspected on Thursday, it's a libtraceevent issue. Once the bad build for libtraceevent gets into sstate, perf will always fail.
As Ross predicted, it is now happening all the time: https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/7618/steps/11/logs/stdio qemumips64 on rocky9-ty-1
$ make -j libs CLEAN plugins CLEAN src CLEAN libtraceevent DESCEND src libtraceevent.a DESCEND src libtraceevent.so COMPILE FPIC kbuffer-parse.o COMPILE FPIC parse-filter.o COMPILE FPIC parse-utils.o COMPILE FPIC event-parse-api.o COMPILE FPIC event-parse-api.o COMPILE FPIC kbuffer-parse.o COMPILE FPIC tep_strerror.o COMPILE FPIC tep_strerror.o COMPILE FPIC parse-utils.o COMPILE FPIC trace-seq.o COMPILE FPIC parse-filter.o COMPILE FPIC trace-seq.o COMPILE FPIC event-plugin.o COMPILE FPIC event-plugin.o COMPILE FPIC event-parse.o COMPILE FPIC event-parse.o COMPILE SHARED LIB libtraceevent.so.1.7.3 BUILD STATIC LIB libtraceevent.a Note that it compiles kbuffer-parse.o _twice_, so one of the library links is probably reading the .o whilst it's being rewritten. I'll switch the recipe to use Meson.
Fix sent, "[PATCH] libtraceevent: build with Meson".
The build I had run and that had been consistently failing has succeeded after applying Ross's patch: https://lore.kernel.org/openembedded-core/20230827195744.3451852-1-ross.burton@arm.com/T/#u
Merged in 5d060725e706476751b0bda8e6d9f3670aa4b7de.