Created attachment 1284 [details] Log file of meta-toolchain build for qemux86 With the master(master:e04e6dab0afa8f72bf001b0cc106cd13b7694a91) of poky, "bitbake meta-toolchain" failed to build sometimes. The issue is not 100% reproducible each time, "bitbake meta-toolchain" can be repeated to reproduce the failure. During our test, both BB_NUMBER_THREADS and PARALLEL_MAKE are set to 24, we tried the toolchain build for beagleboard, routerstationpro, mpc8315e-rdb, atom-pc and qemux86, the build error is reproduced on beagleboard(the 5th build), mpc8315e-rdb(the 10th build), atom-pc(the 3rd build) and qemux86(the 2nd time). Sometimes the issue happens for the 1st time. Following is the partial log, full log is attached(meta-toolchain-fail.log) | log_check: Using /home/b19537/jenkins/workspace/yocto-upstream-toolchain/label/busy/machine/beagleboard/poky/master/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/meta-toolchain/1.0-r7/temp/log.do_populate_sdk.567 as logfile | Logfile is clean | DEBUG: Shell function populate_sdk_image finished | DEBUG: SITE files ['endian-little', 'bit-32', 'arm-common', 'common-linux', 'common-glibc', 'arm-linux', 'arm-linux-gnueabi', 'common'] | DEBUG: Executing shell function create_sdk_files | DEBUG: Shell function create_sdk_files finished | DEBUG: Executing shell function tar_sdk | tar: ./sysroots/x86_64-pokysdk-linux/var/lib/rpm/__db.002: file changed as we read it | DEBUG: Python function do_populate_sdk finished | ERROR: Function failed: tar_sdk (log file is located at /home/b19537/jenkins/workspace/yocto-upstream-toolchain/label/busy/machine/beagleboard/poky/master/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/meta-toolchain/1.0-r7/temp/log.do_populate_sdk.567)
Seems like this is a timing issue, but I can't find any way that do_package_sdk is getting paralleled
I'm testing on our build server with: BB_NUMBER_THREADS = "40" PARALLEL_MAKE = "-j 32" and the build finished just fine 14 times now (and counting). MACHINE is qemux86. Do you have anything else different in your local.conf file besides those 2 variables? FYI: while writing this it got to build no. 17...
Please see attached local.conf. The OS on my build server is CentOS release 5.9.
Created attachment 1291 [details] local.conf file
I wasn't able to reproduce this on Ubuntu Server 12.04 with BB_NUMBER_THREADS = "40" PARALLEL_MAKE = "-j 32" After I tried for qemux86 for 25 times, I switched to atom-pc and left it over night. It ran successfully for 728 times... However, it appears that you're using Jenkins to do builds. Would you please run this oneliner, outside Jenkins environment? i=0; while [ $? = 0 ]; do ((i = i + 1)); echo "################## Run no. $i ################"; MACHINE=atom-pc bitbake meta-toolchain; done At least, you do the same test as I do and establish a baseline. I've read do_populate_sdk() function several times and it doesn't seem to do any parallelism.
I am running the build outside of Jenkins and will post the result when the test finished.
Created attachment 1293 [details] meta-toolchain build log outside of Jenkins
The error appears in the 6th build, please see attached meta-toolchain-build-outside-Jenkins.log for full log.
We had an internal discussion about this issue and, apparently, there used to be some issues with pseudo running on older kernels. Something related to fsync syscalls... I'm CCing Peter Sebach, the pseudo maintainer, maybe he remembers what was the problem and can provide a workaround.
If all that's happening is tar mysteriously failing, and there's something about a file changing as it was being read, the thing we encountered was: On some kernels (not sure whether older, newer, or what), and some filesystems, Linux will apparently update either file size or file mtime LONG after a file is done being written to. This is, obviously, a severe bug. Unfortunately, it's not a bug we can fix; the workaround is to forcibly fsync things. In the WR build system, we were using a modified prelink which would fsync files after writing them, and we had to set the environment variable which prevents pseudo from disabling the fsyncs for this to work.
Zhenhua, any news on this? Were you able to find a workaround base on Peter's suggestions? I think the environment variable Peter was referring to is PSEUDO_ALLOW_FSYNC. Right Peter?
I will give a test with "export PSEUDO_ALLOW_FSYNC=1" and provide the result.
Laurentiu, the issue is reproducible with PSEUDO_ALLOW_FSYNC=1, following is my test command and the issue appears in the second build. i=0; while [ $? = 0 ]; do ((i = i + 1)); echo "################## Run no. $i ################"; PSEUDO_ALLOW_FSYNC=1 MACHINE=qemux86 bitbake meta-toolchain; done | log_check: Using /home/b19537/workspace/poky-oss/build-tc/tmp/work/i586-poky-linux/meta-toolchain/1.0-r7/temp/log.do_populate_sdk.4672 as logfile | Logfile is clean | DEBUG: Shell function populate_sdk_image finished | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] | DEBUG: Executing shell function create_sdk_files | DEBUG: Shell function create_sdk_files finished | DEBUG: Executing shell function tar_sdk | tar: ./sysroots/x86_64-pokysdk-linux/var/lib/rpm/__db.003: file changed as we read it | DEBUG: Python function do_populate_sdk finished
I am not quite sure PSEUDO_ALLOW_FSYNC will work when set globally like that -- because I think it may not be in the whitelist of things that bitbake will pass through from the environment.
If we want an environment variable to be available to tasks, we can do an export PSEUDO_ALLOW_FSYNC in local.conf. This way, PSEUDO_ALLOW_FSYNC will be available to all tasks... But I'm not really sure pseudo will use it. It's worth trying though.
(In reply to comment #15) > If we want an environment variable to be available to tasks, we can do an > > export PSEUDO_ALLOW_FSYNC > > in local.conf. This way, PSEUDO_ALLOW_FSYNC will be available to all > tasks... But I'm not really sure pseudo will use it. It's worth trying > though. I will give a try to export PSEUDO_ALLOW_FSYNC in local.conf.
I added 'export PSEUDO_ALLOW_FSYNC="1"' and 'BB_ENV_EXTRAWHITE_append = " PSEUDO_ALLOW_FSYNC"' in local.conf, the issue appears in the 9th build. $ cat conf/local.conf | grep PSEUDO_ALLOW_FSYNC export PSEUDO_ALLOW_FSYNC="1" BB_ENV_EXTRAWHITE_append = " PSEUDO_ALLOW_FSYNC" $ $ bitbake meta-toolchain -e | grep BB_ENV_EXTRAWHITE # $BB_ENV_EXTRAWHITE [3 operations] BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR PSEUDO_ALLOW_FSYNC" # "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV" BB_HASHCONFIG_WHITELIST="TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH DATE TIME SSH_AGENT_PID SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV"
I suspect you need to both allow the fsync, *and* add a sync call before the tar command. You could probably also do soemthing like: PSEUDO_UNLOAD=1 sync before the tar. It doesn't change the fact that the underlying kernel is *seriously* broken and I'd rather the build system just said so and refused to run.
(In reply to comment #18) > I suspect you need to both allow the fsync, *and* add a sync call before the > tar command. You could probably also do soemthing like: > > PSEUDO_UNLOAD=1 sync I will try the suggestion. > before the tar. It doesn't change the fact that the underlying kernel is > *seriously* broken and I'd rather the build system just said so and refused > to run. The kernel version on my machine is 2.6.18-348.18.1.el5.centos.plus.
I still want to find out what's actually going on in the fairly rare failures we were seeing. In our case, what was doing it specifically was running the prelinker without fsync, and then every so often tar would claim that a file changed while being written. I would like to modify tar to say specifically what it thinks changed, and reproduce that some day. (The underlying problem we used to have with prelink was that the block count used by "du" and friends would change erratically seconds to minutes after a write was complete. Ugh!)
(In reply to comment #19) > (In reply to comment #18) > > I suspect you need to both allow the fsync, *and* add a sync call before the > > tar command. You could probably also do soemthing like: > > > > PSEUDO_UNLOAD=1 sync > I will try the suggestion. > > > before the tar. It doesn't change the fact that the underlying kernel is > > *seriously* broken and I'd rather the build system just said so and refused > > to run. > The kernel version on my machine is 2.6.18-348.18.1.el5.centos.plus. The issue doesn't appear after add PSEUDO_UNLOAD=1 sync before tar.
*** Bug 5287 has been marked as a duplicate of this bug. ***
This isn't something we're going to fix in the public project. Please use a system with a working filesystem. I appreciate they are workaround but they're too nasty to include in the main releases.