Bug 3632

Summary:	[Autobuilder] fri2-noemgd build failed on autobuilder due to stale NFS Handle
Product:	[Infrastructure] AutoBuilder	Reporter:	Andrei Dinu <andrei.adrianx.dinu>
Component:	autobuilder	Assignee:	Michael Halstead <mhalstead>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium	CC:	elizabeth.flanagan, infras.ab.watcher, Infras.watcher, sgw
Version:	unspecified
Target Milestone:	1.4
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	---

Description Andrei Dinu 2012-12-21 10:47:38 UTC

ERROR: Function failed: sstate_create_package (see /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 for further information)
NOTE: Running task 5869 of 7668 (ID: 1947, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-connectivity/avahi/avahi_0.6.31.bb, do_package_write_ipk)
NOTE: Running noexec task 5870 of 7668 (ID: 4762, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb, do_package_write)
NOTE: Running task 5871 of 7668 (ID: 486, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-core/psplash/psplash_git.bb, do_configure)
ERROR: Logfile of failure stored in: /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_populate_sysroot
| DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
| DEBUG: Executing shell function sysroot_stage_all
| DEBUG: Shell function sysroot_stage_all finished
| DEBUG: Python function do_populate_sysroot finished
| DEBUG: Executing python function do_qa_staging
| NOTE: QA checking staging
| DEBUG: Python function do_qa_staging finished
| DEBUG: Executing python function sstate_task_postfunc
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Executing python function do_siteconfig
| DEBUG: Executing python function siteconfig_do_siteconfig
| DEBUG: No site_config directory, skipping do_siteconfig
| DEBUG: Python function siteconfig_do_siteconfig finished
| DEBUG: Python function do_siteconfig finished
| DEBUG: Preparing tree /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir for packaging at /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/sysroot-destdir
| Removing hardcoded paths from sstate package: 'find /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd' | tee /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/fixmepath | xargs --no-run-if-empty sed -i -e 's:/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd:FIXMESTAGINGDIRHOST:g''
| DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
| DEBUG: Executing shell function sstate_create_package
| mv: accessing `/srv/www/vhosts/autobuilder.yoctoproject.org/sstate/21/sstate-rpm-core2-poky-linux-5.4.9-r59-core2-3-2116280e577776a47eb6aa61f39d8446_populate-sysroot.tgz': Stale NFS file handle
| DEBUG: Python function sstate_task_postfunc finished
| ERROR: Function failed: sstate_create_package (see /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 for further information)
NOTE: recipe rpm-5.4.9-r59: task do_populate_sysroot: Failed
ERROR: Task 526 (/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-devtools/rpm/rpm_5.4.9.bb, do_populate_sysroot) failed with exit code '1'

Comment 1 Beth Flanagan 2013-01-08 22:08:25 UTC

We've seen this before, generally during times of high network load. Reassigning to Michael Halstead to see if he has some ideas. It's a generally rare occurance.

Comment 2 Michael Halstead 2013-01-08 22:30:18 UTC

I do not see high network load on nas02. At this point I suspect this is related to the shortcomings of NFSv3. The nas01 is using NFSv4 and as far as I know we don't see these errors there.

nas02 was switched to provide NFSv3 after it was upgraded to OpenSUSE 12.2 from Ubuntu 12.04LTS and idmapd failed to work correctly. Attempts to fix idmapd failed at the time. 

I need to schedule 3 hours of downtime for ab03, ab07, ab08, and ab09 in order to attempt to switch back to NFSv4 after applying all available updates.

Comment 3 Michael Halstead 2013-01-10 22:54:38 UTC

The relevant builders have been upgraded to openSUSE 12.2 and mounts have been switched to NFSv4. I expect this will fix the stale NFS handle problem.