12302 2017-11-06 15:14:17 +0000 kernel's perf compilation failing in nightly-arm-lsb 2018-02-09 14:46:29 +0000 1 1 10 QA/Testing Build Testing general 2.5 x86 Multiple RESOLVED WORKSFORME Medium+ normal 2.5 M3 1 leonardo.sandoval.gonzalez randy.macleod akuster randy.macleod ross.burton --- --- No (bug/feature does not impact docs) oldest_to_newest 77975 0 leonardo.sandoval.gonzalez 2017-11-06 15:14:17 +0000 Seems to be some kind of permission issue when AB is building kernel's perf source code as seen on this code snippet: ------ | BUILD: Doing 'make &#27;[33m-j56&#27;[m' parallel build | | Auto-detecting system features: | ... dwarf: [ &#27;[32mon&#27;[m ] | ... dwarf_getlocations: [ &#27;[32mon&#27;[m ] | ... glibc: [ &#27;[32mon&#27;[m ] | ... gtk2: [ &#27;[31mOFF&#27;[m ] | ... libaudit: [ &#27;[31mOFF&#27;[m ] | ... libbfd: [ &#27;[32mon&#27;[m ] | ... libelf: [ &#27;[32mon&#27;[m ] | ... libnuma: [ &#27;[31mOFF&#27;[m ] | ... numa_num_possible_cpus: [ &#27;[31mOFF&#27;[m ] | ... libperl: [ &#27;[32mon&#27;[m ] | ... libpython: [ &#27;[32mon&#27;[m ] | ... libslang: [ &#27;[32mon&#27;[m ] | ... libcrypto: [ &#27;[32mon&#27;[m ] | ... libunwind: [ &#27;[32mon&#27;[m ] | ... libdw-dwarf-unwind: [ &#27;[32mon&#27;[m ] | ... zlib: [ &#27;[32mo n&#27;[m ]< br>| ... lzma: [ &#27;[32mon&#27;[m ] | ... get_cpuid: [ &#27;[31mOFF&#27;[m ] | ... bpf: [ &#27;[32mon&#27;[m ] | | Makefile.config:352: DWARF support is off, BPF prologue is disabled | Makefile.config:479: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev | GEN /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/common-cmds.h | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep.o | HOSTLD /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep-in.o | LINK /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1 .0/fixde p | MKDIR /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/ | MKDIR /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/ | MKDIR /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/ | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/json.o | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jsmn.o | HOSTCC /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jevents.o | / bin/sh: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0//fixdep: Permission denied | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work-shared/qemuarm/kernel-source/tools/build/Makefile.build:91: recipe for target '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jsmn.o' failed | make[2]: *** [/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm-poky-linux-gnueabi/perf/1.0-r9/perf-1.0/pmu-events/jsmn.o] Error 1 | make[2]: *** Waiting for unfinished jobs.... | /bin/sh: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/qemuarm -------- AB info: Build status: FAILURE Build worker for this Build: opensuse423.yocto.io Complete logs for all build steps: https://autobuilder.yocto.io/builders/nightly-arm-lsb/builds/550 Build Reason: Triggerable(trigger_main-build) Build Properties: • Contents of bblayers.conf (Note. LCONF_VERSION will not show up correctly) POKY_BBLAYERS_CONF_VERSION = BBPATH = "${TOPDIR}" BBFILES ?="" BBLAYERS &#43;= " \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta-poky \ /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta-yocto-bsp \ " • BuildImages : core-image-lsb core-image-lsb-sdk • DEST : /srv/autobuilder/autobuilder.yoctoproject.org/pub/nightly/20171103-3 • DISTRO : poky-lsb • KEEP_BUILD_HIST : False • Contents of auto.conf (local.conf for autobuilders) INHERIT &#43;= "report-error" DISTRO = "poky-lsb" PACKAGE_CLASSES = "package_rpm package_deb package_ipk" BB_NUMBER_THREADS = "24" PARALLEL_MAKE = "-j 28" SDKMACHINE ?= "i686" BB_TASK_NICE_LEVEL = "5" BB_TASK_NICE_LEVEL_task-testimage = "0" BB_TASK_IONICE_LEVEL = "2.7" BB_TASK_IONICE_LEVEL_task-testimage = "2.1" DL_DIR = "/srv/autobuilder/autobuilder.yoctoproject.org/current_sources" SSTATE_DIR ?= "/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstate/" MACHINE = "beaglebone" PREMIRRORS = "" BB_GENERATE_MIRROR_TARBALLS = "1" PRSERV_HOST = "localhost:0" ADTREPO = "http://adtrepo.yoctoproject.org//2.4" QEMU_USE_KVM = "True" SDK_EXT_TYPE = "minimal" SDK_INCLUDE_TOOLCHAIN = "1" SANITY_TESTED_DISTROS = "" 78109 1 ross.burton 2017-11-09 15:54:55 +0000 *** Bug 12325 has been marked as a duplicate of this bug. *** 78176 2 akuster 2017-11-13 16:07:40 +0000 poking around google, I found (Those errors would be caused by not linking against libdw, which it should be doing since the build script sets NO_LIBUNWIND=1) I am planning on trying to reproducing this today, maybe I can help on this issue. 78177 3 bruce.ashfield 2017-11-13 16:32:16 +0000 I haven't done anything with this yet. If you want the bug, feel free to take it :D or just dump your findings here and I can pick things up later. 78178 4 akuster 2017-11-13 16:36:46 +0000 taking. think this was introduce by http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.9/commit/tools/perf/Makefile.config?h=standard/base&id=03f5be20ec9befe477e9978bdba2b1f4f2ca9e42 78218 5 akuster 2017-11-15 17:57:58 +0000 can not reproduce this with latest master. 78219 6 akuster 2017-11-15 18:01:22 +0000 (In reply to comment #5) > can not reproduce this with latest master. can we re-launch just this build? 78221 7 ross.burton 2017-11-15 18:17:52 +0000 It's a race so won't reproduce on demand unless you do the right make invocations. 78353 8 randy.macleod 2017-11-27 17:06:00 +0000 https://autobuilder.yocto.io/builders/nightly-arm-lsb?numbuilds=60 shows 8 successful builds in a row. I guess this just shows how infrequent the bug is. 78380 9 randy.macleod 2017-11-28 17:27:04 +0000 I ran an overnight check and I'm NOT able to duplicate the bug on a 128 core) build host even when running lots of cpu/io activity using stress in parallel with a loop of: bitbake -c cleanall perf linux-yocto && bitbake perf linux-yocto I checked upstream in the main linux git repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git 4fbd8d194f06 (HEAD -> master, tag: v4.15-rc1, origin/master, origin/HEAD) Linux 4.15-rc1 and I did NOT see a fix. (git log tools/perf) We now have 10 successful builds in a row: https://autobuilder.yocto.io/builders/nightly-arm-lsb?numbuilds=50 Has anything changed in the build infrastructure that is related to this defect? 78396 10 ross.burton 2017-11-29 09:08:53 +0000 No, it's just a race. None of the recent ab builds have been touching anything that would cause perf to rebuild so I wouldn't take that as an assurance that it's fixed. 78823 11 randy.macleod 2017-12-22 21:45:39 +0000 Are you saying that *all* of the nightly builds use sstate-cache? How often does the system run a full re-build without sstate-cache? I would hope that your reply is once a week or better still every night for half of the build machines since there may be value testing full sstate-cache systems as well. I still haven't found a nightly-arm build that failed on perf. Am I not looking in the right place? Most of the recent nightly-arm failures are from esdk failures such as: https://autobuilder.yocto.io/builders/nightly-arm/builds/703/steps/Running%20Sanity%20Tests/logs/stdio FAIL: test_kernel_module (kernelmodule.KernelModuleTest) and: https://autobuilder.yocto.io/builders/nightly-arm/builds/716/steps/BuildImages_2/logs/stdio ERROR: buildtools-tarball-1.0-r0 do_populate_sdk: Could not invoke dnf. ¯\_(ツ)_/¯ 79002 12 akuster 2018-01-10 19:23:42 +0000 has anyone seen this issue since? 79013 13 randy.macleod 2018-01-10 23:30:01 +0000 No sign of it on our build cluster. Nothing in kernel.org it seemed either so I had a quick look at the perf recipe and found: ... PACKAGECONFIG[libunwind] = ",NO_LIBUNWIND=1 NO_LIBDW_DWARF_UNWIND=1,libunwind" PACKAGECONFIG[libnuma] = ",NO_LIBNUMA=1" PACKAGECONFIG[systemtap] = ",NO_SDT=1,systemtap" In looking at the perf Makefiles, I see: ifndef NO_LIBAUDIT ifneq ($(feature-libaudit), 1) msg := $(warning No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev); NO_LIBAUDIT := 1 else CFLAGS += -DHAVE_LIBAUDIT_SUPPORT EXTLIBS += -laudit $(call detected,CONFIG_AUDIT) endif endif and as Ross said, it seems like a race (or perhaps host contamination). I'm working on trying to manually reproduce the error then I'll add the: PACKAGECONFIG[audit] = ",NO_LIBAUDIT=1, audit" line and send a patch if that works. 79069 14 randy.macleod 2018-01-15 03:43:20 +0000 No that's wrong, the libaudit warning is a separate problem. I've opened an enhancement: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12482 to track it. The fixdep permissions build error must have happened before the 4.12 kernel update since a commit: abb26210a395 perf tools: Force fixdep compilation at the start of the build from Dec 2016, is in the 4.12 kernel tree. $ git tag --contains abb26210a39522a6645bce3f438ed9a26bedb11b | grep -v rc v4.10 v4.11 v4.12 v4.13 v4.14 *With* the commit, fixdep is built before the auto-detection step: ---------------- make: Entering directory '/.../tmp-glibc/work-shared/qemuarm/kernel-source/tools/perf' BUILD: Doing 'make -j128' parallel build HOSTCC /.../tmp-glibc/work/qemuarm-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep.o HOSTLD /.../tmp-glibc/work/qemuarm-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep-in.o LINK /.../tmp-glibc/work/qemuarm-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/fixdep Auto-detecting system features: ... -------------------------------- whereas the logs given: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12302#c0 start with auto-detection. Can anyone confirm that this is only happening on pyro and earlier? If so, should we backport the fix? It did not apply cleanly to 4.9.49 so best to give the defect to someone else. 79403 15 randy.macleod 2018-02-09 14:46:29 +0000 There was a race condition with the build but as explained already, this has been resolved for the 2.4 and master branches. I don't think it's worth backporting the fix since it's relatively rare so I'm closing this defect.