Seems to be some kind of permission issue when AB is building kernel's perf source code as seen on this code snippet: ------ | BUILD: Doing 'make [33m-j56[m' parallel build | | Auto-detecting system features: | ... dwarf: [ [32mon[m ] | ... dwarf_getlocations: [ [32mon[m ] | ... glibc: [ [32mon[m ] | ... gtk2: [ [31mOFF[m ] | ... libaudit: [ [31mOFF[m ] | ... libbfd: [ [32mon[m ] | ... libelf: [ [32mon[m ] | ... libnuma: [ [31mOFF[m ] | ... numa_num_possible_cpus: [ [31mOFF[m ] | ... libperl: [ [32mon[m ] | ... libpython: [ [32mon[m ] | ... libslang: [ [32mon[m ] | ... libcrypto: [ [32mon[m ] | ... libunwind: [ [32mon[m ] | ... libdw-dwarf-unwind: [ [32mon[m ] | ... zlib: [ [32mo n[m ]< br>| ... lzma: [ [32mon[m ] | ... get_cpuid: [ [31mOFF[m ] | ... bpf: [ [32mon[m ] | | Makefile.config:352: DWARF support is off, BPF prologue is disabled | Makefile.config:479: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev | GEN /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/common-cmds.h | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep.o | HOSTLD /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep-in.o | LINK /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1 .0/fixde p | MKDIR /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/ | MKDIR /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/ | MKDIR /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/ | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/json.o | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jsmn.o | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jevents.o | / bin/sh: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0//fixdep: Permission denied | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work-shared/qemuarm/kernel-source/tools/build/Makefile.build:91: recipe for target '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jsmn.o' failed | make[2]: *** [/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jsmn.o] Error 1 | make[2]: *** Waiting for unfinished jobs.... | /bin/sh: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm -------- AB info: Build status: FAILURE Build worker for this Build: opensuse423.yocto.io Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-arm-lsb/builds/550 Build Reason: Triggerable(trigger_main-build) Build Properties: • Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly) POKY_BBLAYERS_CONF_VERSION = BBPATH = "${TOPDIR}" BBFILES ?="" BBLAYERS += " \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta-poky \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta-yocto-bsp \ " • BuildImages : core-image-lsb core-image-lsb-sdk • DEST : /srv/autobuilder/autobuilder.yoctoproject.org/pub/nightly/20171103-3 • DISTRO : poky-lsb • KEEP_BUILD_HIST : False • Contents of auto.conf (local.conf for autobuilders) INHERIT += "report-error" DISTRO = "poky-lsb" PACKAGE_CLASSES = "package_rpm package_deb package_ipk" BB_NUMBER_THREADS = "24" PARALLEL_MAKE = "-j 28" SDKMACHINE ?= "i686" BB_TASK_NICE_LEVEL = "5" BB_TASK_NICE_LEVEL_task-testimage = "0" BB_TASK_IONICE_LEVEL = "2.7" BB_TASK_IONICE_LEVEL_task-testimage = "2.1" DL_DIR = "/srv/autobuilder/autobuilder.yoctoproject.org/current_sources" SSTATE_DIR ?= "/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstate/" MACHINE = "beaglebone" PREMIRRORS = "" BB_GENERATE_MIRROR_TARBALLS = "1" PRSERV_HOST = "localhost:0" ADTREPO = "http://adtrepo.yoctoproject.org//2.4" QEMU_USE_KVM = "True" SDK_EXT_TYPE = "minimal" SDK_INCLUDE_TOOLCHAIN = "1" SANITY_TESTED_DISTROS = ""
*** Bug 12325 has been marked as a duplicate of this bug. ***
poking around google, I found (Those errors would be caused by not linking against libdw, which it should be doing since the build script sets NO_LIBUNWIND=1) I am planning on trying to reproducing this today, maybe I can help on this issue.
I haven't done anything with this yet. If you want the bug, feel free to take it :D or just dump your findings here and I can pick things up later.
taking. think this was introduce by http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.9/commit/tools/perf/Makefile.config?h=standard/base&id=03f5be20ec9befe477e9978bdba2b1f4f2ca9e42
can not reproduce this with latest master.
(In reply to comment #5) > can not reproduce this with latest master. can we re-launch just this build?
It's a race so won't reproduce on demand unless you do the right make invocations.
https://autobuilder.yocto.io/builders/nightly-arm-lsb?numbuilds=60 shows 8 successful builds in a row. I guess this just shows how infrequent the bug is.
I ran an overnight check and I'm NOT able to duplicate the bug on a 128 core) build host even when running lots of cpu/io activity using stress in parallel with a loop of: bitbake -c cleanall perf linux-yocto && bitbake perf linux-yocto I checked upstream in the main linux git repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git 4fbd8d194f06 (HEAD -> master, tag: v4.15-rc1, origin/master, origin/HEAD) Linux 4.15-rc1 and I did NOT see a fix. (git log tools/perf) We now have 10 successful builds in a row: https://autobuilder.yocto.io/builders/nightly-arm-lsb?numbuilds=50 Has anything changed in the build infrastructure that is related to this defect?
No, it's just a race. None of the recent ab builds have been touching anything that would cause perf to rebuild so I wouldn't take that as an assurance that it's fixed.
Are you saying that *all* of the nightly builds use sstate-cache? How often does the system run a full re-build without sstate-cache? I would hope that your reply is once a week or better still every night for half of the build machines since there may be value testing full sstate-cache systems as well. I still haven't found a nightly-arm build that failed on perf. Am I not looking in the right place? Most of the recent nightly-arm failures are from esdk failures such as: https://autobuilder.yocto.io/builders/nightly-arm/builds/703/steps/Running%20Sanity%20Tests/logs/stdio FAIL: test_kernel_module (kernelmodule.KernelModuleTest) and: https://autobuilder.yocto.io/builders/nightly-arm/builds/716/steps/BuildImages_2/logs/stdio ERROR: buildtools-tarball-1.0-r0 do_populate_sdk: Could not invoke dnf. ¯\_(ツ)_/¯
has anyone seen this issue since?
No sign of it on our build cluster. Nothing in kernel.org it seemed either so I had a quick look at the perf recipe and found: ... PACKAGECONFIG[libunwind] = ",NO_LIBUNWIND=1 NO_LIBDW_DWARF_UNWIND=1,libunwind" PACKAGECONFIG[libnuma] = ",NO_LIBNUMA=1" PACKAGECONFIG[systemtap] = ",NO_SDT=1,systemtap" In looking at the perf Makefiles, I see: ifndef NO_LIBAUDIT ifneq ($(feature-libaudit), 1) msg := $(warning No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev); NO_LIBAUDIT := 1 else CFLAGS += -DHAVE_LIBAUDIT_SUPPORT EXTLIBS += -laudit $(call detected,CONFIG_AUDIT) endif endif and as Ross said, it seems like a race (or perhaps host contamination). I'm working on trying to manually reproduce the error then I'll add the: PACKAGECONFIG[audit] = ",NO_LIBAUDIT=1, audit" line and send a patch if that works.
No that's wrong, the libaudit warning is a separate problem. I've opened an enhancement: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12482 to track it. The fixdep permissions build error must have happened before the 4.12 kernel update since a commit: abb26210a395 perf tools: Force fixdep compilation at the start of the build from Dec 2016, is in the 4.12 kernel tree. $ git tag --contains abb26210a39522a6645bce3f438ed9a26bedb11b | grep -v rc v4.10 v4.11 v4.12 v4.13 v4.14 *With* the commit, fixdep is built before the auto-detection step: ---------------- make: Entering directory '/.../tmp-glibc/work-shared/qemuarm/kernel-source/tools/perf' BUILD: Doing 'make -j128' parallel build HOSTCC /.../tmp-glibc/work/qemuarm-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep.o HOSTLD /.../tmp-glibc/work/qemuarm-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep-in.o LINK /.../tmp-glibc/work/qemuarm-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep Auto-detecting system features: ... -------------------------------- whereas the logs given: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12302#c0 start with auto-detection. Can anyone confirm that this is only happening on pyro and earlier? If so, should we backport the fix? It did not apply cleanly to 4.9.49 so best to give the defect to someone else.
There was a race condition with the build but as explained already, this has been resolved for the 2.4 and master branches. I don't think it's worth backporting the fix since it's relatively rare so I'm closing this defect.