We see a strange failure on the Autobuilder when pushing to a buildhistory repo fails: error: src refspec poky/master/nightly-x86-64-lsb does not match any. error: failed to push some refs to 'file:////srv/autobuilder/tmp/yocto-autobuilder/buildhistory-repo' Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/bitbake/lib/bb/ui/knotty.py", line 426, in main helper.eventHandler(event) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/bitbake/lib/bb/ui/uihelper.py", line 39, in eventHandler del self.running_tasks[event.pid] KeyError: 5463 It seems likely that the buildhistory failure is causing a pid to be reaped twice, only the buildhistory shouldn't be generating an event (it's an eventhandler, not a task) so it's unclear why eventHandler() in uihelper is being called with the pid and an event which causes reaping more than once. Perhaps a bogus event is somehow being generated?
Full log (currently) available at: http://autobuilder.yocto.io:8010/builders/nightly-x86-64-lsb/builds/50/steps/BuildImages/logs/stdio
http://autobuilder.yocto.io:8010/builders/nightly-mips64/builds/38/steps/BuildImages/logs/stdio suggests that it isn't buildhistory specific.
Both logs have something like this in: NOTE: Sending SIGTERM to remaining 32 tasks Which is from bitbake-worker:handle_finishnow(), called via either sigterm_exception() or the finishnow event being passed over the queue from the cooker. Were these builds both aborted in the autobuilder UI? Moving to bitbake. Richard, I'll carry on digging tomorrow but do you have any thoughts?
We just saw two more of these failures on the new AB cluster: http://autobuilder.yocto.io:8010/builders/nightly-no-x11/builds/59/steps/BuildImages/logs/stdio http://autobuilder.yocto.io:8010/builders/nightly-arm/builds/115/steps/BuildImages/logs/stdio
For the record, no stop command was issued and there are no buildhistory failures I can see in these cases. What is also odd is the logging is corrupted before the SIGTERM message with task logging being visible. Not sure why.
NOTE: Running task 4053 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-connectivity/nfs-utils/libnfsidmap_0.25.bb:do_install) NOTE: recipe orc-0.4.25-r0: task do_package: Started NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_install: Succeeded NOTE: Running task 4054 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-graphics/xorg-proto/scrnsaverproto_1.2.2.bb:do_populate_sysroot) NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_populate_sysroot: Started NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libusb-compat/1_0.1.5-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libusb-compat/1_0.1.5-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/xf86dgaproto/1_2.1-r2/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/xf86dgaproto/1_2.1-r2/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/gmp/6.1.1-r0/sstate-build-packagedata/ \( -name "*.la" -o -name "*-config" -o -name "*_config" -o -name "gmp.h" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/gmp/6.1.1-r0/sstate-build-packagedata/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Sending SIGTERM to remaining 4 tasks NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/speexdsp/1.2rc3-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/speexdsp/1.2rc3-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/orc/0.4.25-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/orc/0.4.25-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: make -j 48 DESTDIR=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/tremor/20150107-r1/image install NOTE: make -j 48 DESTDIR=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libpcre/8.39-r0/image install This is what the autobuilder stdout says: NOTE: make -j 48 -e MAKEFLAGS= INSTALL_PREFIX=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/openssl/1.0.2j-r0/image MANDIR=/usr/share/man install NOTE: make -j 48 DESTDIR=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/libevent/2.0.22-r0/image install NOTE: Using umask 0o002 (not 22) for sstate packaging NOTE: Removing hardcoded paths from sstate package: 'find /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/resourceproto/1_1.2.0-r1/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm' | tee /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/armv5e-poky-linux-gnueabi/resourceproto/1_1.2.0-r1/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/sysroots/qemuarm:FIXMESTAGINGDIRHOST:g'' NOTE: kmod: Rename (link) /sbin/insmod -> /sbin/insmod.kmod NOTE: kmod: Rename (link) /sbin/modprobe -> /sbin/modprobe.kmod NOTE: kmod: Rename (link) /bin/lsmod -> /bin/lsmod.kmod NOTE: kmod: Rename (link) /sbin/depmod -> /sbin/depmod.kmod NOTE: kmod: Rename (link) /sbin/rmmod -> /sbin/rmmod.kmod NOTE: kmod: Rename (link) /sbin/modinfo -> /sbin/modinfo.kmod NOTE: adding update-alternatives calls to postinst/prerm for kmod NOTE: update-alternatives --install /sbin/insmod insmod /sbin/insmod.kmod 60 update-alternatives --install /sbin/modprobe modprobe /sbin/modprobe THe bitbake console log just ends with: NOTE: recipe presentproto-1.0+gitAUTOINC+24f3a56e54-r1: task do_install: Started NOTE: recipe xf86dgaproto-1_2.1-r2: task do_package: Started NOTE: recipe mtdev-1.1.5-r0: task do_compile: Succeeded NOTE: Running task 4052 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-graphics/wayland/mtdev_1.1.5.bb:do_install) NOTE: recipe libnl-1_3.2.28-r0: task do_package: Started NOTE: recipe speexdsp-1.2rc3-r0: task do_package: Started NOTE: recipe libnfsidmap-0.25-r0: task do_compile: Succeeded NOTE: Running task 4053 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-connectivity/nfs-utils/libnfsidmap_0.25.bb:do_install) NOTE: recipe orc-0.4.25-r0: task do_package: Started NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_install: Succeeded NOTE: Running task 4054 of 7282 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-graphics/xorg-proto/scrnsaverproto_1.2.2.bb:do_populate_sysroot) NOTE: recipe scrnsaverproto-1_1.2.2-r0: task do_populate_sysroot: Started so for some reason, no further logs are logged.
Also, nothing abnormal in the scrnsaverproto do_populate_sysroot logfile.
We haven't seen this for ages, if it happens again we can reopen...