10271 2016-09-13 20:12:12 +0000 KeyError exception when pushing buildhistory fails 2017-03-20 17:54:35 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 2.2 x86 Multiple RESOLVED WORKSFORME Medium+ normal 2.3 M4 1 joshuagloe ross.burton meta.mr.watcher meta.watcher poky.bs.watcher poky.watcher richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 66104 0 joshuagloe 2016-09-13 20:12:12 +0000 We see a strange failure on the Autobuilder when pushing to a buildhistory repo fails: error: src refspec poky/master/nightly-x86-64-lsb does not match any. error: failed to push some refs to 'file:////srv/autobuilder/tmp/yocto-autobuilder/buildhistory-repo' Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/bitbake/lib/bb/ui/knotty.py", line 426, in main helper.eventHandler(event) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/bitbake/lib/bb/ui/uihelper.py", line 39, in eventHandler del self.running_tasks[event.pid] KeyError: 5463 It seems likely that the buildhistory failure is causing a pid to be reaped twice, only the buildhistory shouldn't be generating an event (it's an eventhandler, not a task) so it's unclear why eventHandler() in uihelper is being called with the pid and an event which causes reaping more than once. Perhaps a bogus event is somehow being generated? 66105 1 joshuagloe 2016-09-13 20:13:03 +0000 Full log (currently) available at: http://autobuilder.yocto.io:8010/builders/nightly-x86-64-lsb/builds/50/steps/BuildImages/logs/stdio 66165 2 ross.burton 2016-09-14 23:17:52 +0000 http://autobuilder.yocto.io:8010/builders/nightly-mips64/builds/38/steps/BuildImages/logs/stdio suggests that it isn't buildhistory specific. 66168 3 ross.burton 2016-09-14 23:26:05 +0000 Both logs have something like this in: NOTE: Sending SIGTERM to remaining 32 tasks Which is from bitbake-worker:handle_finishnow(), called via either sigterm_exception() or the finishnow event being passed over the queue from the cooker. Were these builds both aborted in the autobuilder UI? Moving to bitbake. Richard, I'll carry on digging tomorrow but do you have any thoughts? 67032 4 richard.purdie 2016-10-07 07:23:16 +0000 We just saw two more of these failures on the new AB cluster: http://autobuilder.yocto.io:8010/builders/nightly-no-x11/builds/59/steps/BuildImages/logs/stdio http://autobuilder.yocto.io:8010/builders/nightly-arm/builds/115/steps/BuildImages/logs/stdio 67033 5 richard.purdie 2016-10-07 07:24:36 +0000 For the record, no stop command was issued and there are no buildhistory failures I can see in these cases. What is also odd is the logging is corrupted before the SIGTERM message with task logging being visible. Not sure why. 67042 6 richard.purdie 2016-10-07 16:06:53 +0000 NOTE: Running task 4053 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-connectivity/nfs-utils/libnfsidmap_0.25.bb:do_install) NOTE: recipe orc-0.4.25-r0: task do_package: Started NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_install: Succeeded NOTE: Running task 4054 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-graphics/xorg-proto/scrnsaverproto_1.2.2.bb:do_populate_sysroot) NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_populate_sysroot: Started NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libusb-compat/1_0.1.5-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libusb-compat/1_0.1.5-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/xf86dgaproto/1_2.1-r2/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/xf86dgaproto/1_2.1-r2/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/gmp/6.1.1-r0/sstate-build-packagedata/ \( -name "*.la" -o -name "*-config" -o -name "*_config" -o -name "gmp.h" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/gmp/6.1.1-r0/sstate-build-packagedata/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Sending SIGTERM to remaining 4 tasks NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/speexdsp/1.2rc3-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/speexdsp/1.2rc3-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/orc/0.4.25-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/orc/0.4.25-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: make -j 48 DESTDIR=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/tremor/20150107-r1/image install NOTE: make -j 48 DESTDIR=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libpcre/8.39-r0/image install This is what the autobuilder stdout says: NOTE: make -j 48 -e MAKEFLAGS= INSTALL_PREFIX=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/openssl/1.0.2j-r0/image MANDIR=/usr/share/man install NOTE: make -j 48 DESTDIR=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libevent/2.0.22-r0/image install NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/resourceproto/1_1.2.0-r1/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/resourceproto/1_1.2.0-r1/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: kmod: Rename (link) /sbin/insmod -> /sbin/insmod.kmod NOTE: kmod: Rename (link) /sbin/modprobe -> /sbin/modprobe.kmod NOTE: kmod: Rename (link) /bin/lsmod -> /bin/lsmod.kmod NOTE: kmod: Rename (link) /sbin/depmod -> /sbin/depmod.kmod NOTE: kmod: Rename (link) /sbin/rmmod -> /sbin/rmmod.kmod NOTE: kmod: Rename (link) /sbin/modinfo -> /sbin/modinfo.kmod NOTE: adding update-alternatives calls to postinst/prerm for kmod NOTE: update-alternatives --install /sbin/insmod insmod /sbin/insmod.kmod 60 update-alternatives --install /sbin/modprobe modprobe /sbin/modprobe THe bitbake console log just ends with: NOTE: recipe presentproto-1.0+gitAUTOINC+24f3a56e54-r1: task do_install: Started NOTE: recipe xf86dgaproto-1_2.1-r2: task do_package: Started NOTE: recipe mtdev-1.1.5-r0: task do_compile: Succeeded NOTE: Running task 4052 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-graphics/wayland/mtdev_1.1.5.bb:do_install) NOTE: recipe libnl-1_3.2.28-r0: task do_package: Started NOTE: recipe speexdsp-1.2rc3-r0: task do_package: Started NOTE: recipe libnfsidmap-0.25-r0: task do_compile: Succeeded NOTE: Running task 4053 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-connectivity/nfs-utils/libnfsidmap_0.25.bb:do_install) NOTE: recipe orc-0.4.25-r0: task do_package: Started NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_install: Succeeded NOTE: Running task 4054 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-graphics/xorg-proto/scrnsaverproto_1.2.2.bb:do_populate_sysroot) NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_populate_sysroot: Started so for some reason, no further logs are logged. 67045 7 richard.purdie 2016-10-07 16:11:43 +0000 Also, nothing abnormal in the scrnsaverproto do_populate_sysroot logfile. 71471 8 ross.burton 2017-03-20 17:54:35 +0000 We haven't seen this for ages, if it happens again we can reopen...