Bug 3632 - [Autobuilder] fri2-noemgd build failed on autobuilder due to stale NFS Handle
Summary: [Autobuilder] fri2-noemgd build failed on autobuilder due to stale NFS Handle
Status: RESOLVED FIXED
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.4
Assignee: Michael Halstead
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2012-12-21 10:47 UTC by Andrei Dinu
Modified: 2013-01-10 22:54 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Andrei Dinu 2012-12-21 10:47:38 UTC 
ERROR: Function failed: sstate_create_package (see /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 for further information)
NOTE: Running task 5869 of 7668 (ID: 1947, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-connectivity/avahi/avahi_0.6.31.bb, do_package_write_ipk)
NOTE: Running noexec task 5870 of 7668 (ID: 4762, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb, do_package_write)
NOTE: Running task 5871 of 7668 (ID: 486, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-core/psplash/psplash_git.bb, do_configure)
ERROR: Logfile of failure stored in: /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_populate_sysroot
| DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
| DEBUG: Executing shell function sysroot_stage_all
| DEBUG: Shell function sysroot_stage_all finished
| DEBUG: Python function do_populate_sysroot finished
| DEBUG: Executing python function do_qa_staging
| NOTE: QA checking staging
| DEBUG: Python function do_qa_staging finished
| DEBUG: Executing python function sstate_task_postfunc
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd
| DEBUG: Executing python function do_siteconfig
| DEBUG: Executing python function siteconfig_do_siteconfig
| DEBUG: No site_config directory, skipping do_siteconfig
| DEBUG: Python function siteconfig_do_siteconfig finished
| DEBUG: Python function do_siteconfig finished
| DEBUG: Preparing tree /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir for packaging at /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/sysroot-destdir
| Removing hardcoded paths from sstate package: 'find /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd' | tee /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/fixmepath | xargs --no-run-if-empty sed -i -e 's:/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd:FIXMESTAGINGDIRHOST:g''
| DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
| DEBUG: Executing shell function sstate_create_package
| mv: accessing `/srv/www/vhosts/autobuilder.yoctoproject.org/sstate/21/sstate-rpm-core2-poky-linux-5.4.9-r59-core2-3-2116280e577776a47eb6aa61f39d8446_populate-sysroot.tgz': Stale NFS file handle
| DEBUG: Python function sstate_task_postfunc finished
| ERROR: Function failed: sstate_create_package (see /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 for further information)
NOTE: recipe rpm-5.4.9-r59: task do_populate_sysroot: Failed
ERROR: Task 526 (/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-devtools/rpm/rpm_5.4.9.bb, do_populate_sysroot) failed with exit code '1'
Comment 1 Beth Flanagan 2013-01-08 22:08:25 UTC 
We've seen this before, generally during times of high network load. Reassigning to Michael Halstead to see if he has some ideas. It's a generally rare occurance.
Comment 2 Michael Halstead 2013-01-08 22:30:18 UTC 
I do not see high network load on nas02. At this point I suspect this is related to the shortcomings of NFSv3. The nas01 is using NFSv4 and as far as I know we don't see these errors there.

nas02 was switched to provide NFSv3 after it was upgraded to OpenSUSE 12.2 from Ubuntu 12.04LTS and idmapd failed to work correctly. Attempts to fix idmapd failed at the time. 

I need to schedule 3 hours of downtime for ab03, ab07, ab08, and ab09 in order to attempt to switch back to NFSv4 after applying all available updates.
Comment 3 Michael Halstead 2013-01-10 22:54:38 UTC 
The relevant builders have been upgraded to openSUSE 12.2 and mounts have been switched to NFSv4. I expect this will fix the stale NFS handle problem.