Bug 7101

Summary: Race condition populating site files - tasks getting re-executed after sysroot setscene
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Paul Eggleton <bluelightning>
Component: devtools / tool chainAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: meta.mr.watcher, meta.watcher, richard.purdie
Version: 1.7   
Target Milestone: 1.8   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Paul Eggleton 2014-12-23 10:48:58 UTC 
A string of do_configure failures occurred during the nightly-qa-extras build on the autobuilder:

----------------------- snip ------------------------
ERROR: Function failed: do_configure (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/base-passwd/3.5.29-r0/temp/log.do_configure.26397)
ERROR: Function failed: do_configure (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/temp/log.do_configure.26372)
ERROR: Function failed: do_configure (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/xcb-proto/1.11-r0/temp/log.do_configure.26406)
NOTE: recipe zlib-1.2.8-r0: task do_install_ptest_base: Started
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/temp/log.do_configure.26372
Log data follows:
| DEBUG: Executing python function sysroot_cleansstate
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/sysroot-providers/util-macros
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/aclocal/xorg-macros.m4
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/util-macros/INSTALL
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/pkgconfig/xorg-macros.pc
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/util-macros/
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/pkgconfig/
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/aclocal/
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/sysroot-providers/
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/
| DEBUG: Removing manifest: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/
| DEBUG: Python function sysroot_cleansstate finished
| DEBUG: SITE files ['endian-little', 'common-linux', 'common-glibc', 'bit-64', 'x86_64-linux', 'common']
| DEBUG: Executing shell function autotools_preconfigure
| DEBUG: Shell function autotools_preconfigure finished
| DEBUG: Executing python function autotools_copy_aclocals
| DEBUG: Python function autotools_copy_aclocals finished
| DEBUG: Executing shell function do_configure
| automake (GNU automake) 1.14.1
| Copyright (C) 2013 Free Software Foundation, Inc.
| License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| 
| Written by Tom Tromey <tromey@redhat.com>
|        and Alexandre Duret-Lutz <adl@gnu.org>.
| AUTOV is 1.14
| NOTE: Executing ACLOCAL="aclocal --system-acdir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/build/aclocal-copy/ --automake-acdir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/x86_64-linux/usr/share/aclocal-1.14" autoreconf --verbose --install --force --exclude=autopoint -I /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/util-macros-1.19.0/
| autoreconf: Entering directory `.'
| autoreconf: configure.ac: not using Gettext
| autoreconf: running: aclocal --system-acdir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/build/aclocal-copy/ --automake-acdir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/x86_64-linux/usr/share/aclocal-1.14 -I /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/util-macros-1.19.0/ -I /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/util-macros-1.19.0/ --force
| autoreconf: configure.ac: tracing
| autoreconf: configure.ac: not using Libtool
| autoreconf: running: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/x86_64-linux/usr/bin/autoconf --include=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/util-macros-1.19.0/ --force
| autoreconf: configure.ac: not using Autoheader
| autoreconf: running: automake --add-missing --copy --force-missing
| autoreconf: running: gnu-configize
| autoreconf: Leaving directory `.'
| NOTE: Running /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/util-macros-1.19.0/configure  --build=x86_64-linux 		  --host=x86_64-poky-linux 		  --target=x86_64-poky-linux 		  --prefix=/usr 		  --exec_prefix=/usr 		  --bindir=/usr/bin 		  --sbindir=/usr/sbin 		  --libexecdir=/usr/lib/util-macros 		  --datadir=/usr/share 		  --sysconfdir=/etc 		  --sharedstatedir=/com 		  --localstatedir=/var 		  --libdir=/usr/lib 		  --includedir=/usr/include 		  --oldincludedir=/usr/include 		  --infodir=/usr/share/info 		  --mandir=/usr/share/man 		  --disable-silent-rules 		  --disable-dependency-tracking 		  --with-libtool-sysroot=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64
| configure: WARNING: unrecognized options: --disable-dependency-tracking, --with-libtool-sysroot
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/meta/site/endian-little
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/meta/site/common-linux
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/meta/site/common-glibc
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/meta/site/x86_64-linux
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/meta/site/common
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/x86_64-poky-linux_config_site.d/glibc_config
| configure: loading site script /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/x86_64-poky-linux_config_site.d/ncurses_config
| /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/util-macros-1.19.0/configure: line 1610: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/sysroots/qemux86-64/usr/share/x86_64-poky-linux_config_site.d/ncurses_config: No such file or directory
| Configure failed. The contents of all config.log files follows to aid debugging
| ERROR: oe_runconf failed
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_configure (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-extras/build/build/tmp/work/core2-64-poky-linux/util-macros/1_1.19.0-r0/temp/log.do_configure.26372)
NOTE: recipe util-macros-1_1.19.0-r0: task do_configure: Failed
----------------------- snip ------------------------

This is likely to be some kind of race condition populating the site files. All of the failures were about missing tmp/sysroots/qemux86-64/usr/share/x86_64-poky-linux_config_site.d/ncurses_config.

Full log:

https://autobuilder.yoctoproject.org/main/builders/nightly-qa-extras/builds/136/steps/BuildImages/logs/stdio

(This was for master-next, but the same issue will be present on master.)
Comment 1 Paul Eggleton 2014-12-23 10:51:03 UTC 
I noticed something else which may or may not be related - in the log you can see ncurses do_populate_sysroot_setscene ran and then it went ahead and ran the real task later on, which in theory shouldn't happen. (I've seen this kind of thing before, but I thought the underlying issue might have been fixed already.)
Comment 2 Saul Wold 2014-12-24 18:39:40 UTC 
There are a number of tasks that seem to get rebuilt even though they have been installed via do_populate_sysroot_setscene, possibly an issue with stamps?