3632 2012-12-21 10:47:38 +0000 [Autobuilder] fri2-noemgd build failed on autobuilder due to stale NFS Handle 2013-01-10 22:54:38 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple RESOLVED FIXED Medium normal 1.4 1 andrei.adrianx.dinu mhalstead elizabeth.flanagan infras.ab.watcher Infras.watcher sgw --- --- --- oldest_to_newest 28423 0 andrei.adrianx.dinu 2012-12-21 10:47:38 +0000 ERROR: Function failed: sstate_create_package (see /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 for further information) NOTE: Running task 5869 of 7668 (ID: 1947, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-connectivity/avahi/avahi_0.6.31.bb, do_package_write_ipk) NOTE: Running noexec task 5870 of 7668 (ID: 4762, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb, do_package_write) NOTE: Running task 5871 of 7668 (ID: 486, /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-core/psplash/psplash_git.bb, do_configure) ERROR: Logfile of failure stored in: /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 Log data follows: | DEBUG: Executing python function sstate_task_prefunc | DEBUG: Python function sstate_task_prefunc finished | DEBUG: Executing python function do_populate_sysroot | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] | DEBUG: Executing shell function sysroot_stage_all | DEBUG: Shell function sysroot_stage_all finished | DEBUG: Python function do_populate_sysroot finished | DEBUG: Executing python function do_qa_staging | NOTE: QA checking staging | DEBUG: Python function do_qa_staging finished | DEBUG: Executing python function sstate_task_postfunc | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Staging files from /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir to /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd | DEBUG: Executing python function do_siteconfig | DEBUG: Executing python function siteconfig_do_siteconfig | DEBUG: No site_config directory, skipping do_siteconfig | DEBUG: Python function siteconfig_do_siteconfig finished | DEBUG: Python function do_siteconfig finished | DEBUG: Preparing tree /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sysroot-destdir for packaging at /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/sysroot-destdir | Removing hardcoded paths from sstate package: 'find /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/ \( -name "*.la" -o -name "*-config" -o -name "*_config" \) -type f | xargs grep -l -e '/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd' | tee /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/sstate-build-populate-sysroot/fixmepath | xargs --no-run-if-empty sed -i -e 's:/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/sysroots/fri2-noemgd:FIXMESTAGINGDIRHOST:g'' | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] | DEBUG: Executing shell function sstate_create_package | mv: accessing `/srv/www/vhosts/autobuilder.yoctoproject.org/sstate/21/sstate-rpm-core2-poky-linux-5.4.9-r59-core2-3-2116280e577776a47eb6aa61f39d8446_populate-sysroot.tgz': Stale NFS file handle | DEBUG: Python function sstate_task_postfunc finished | ERROR: Function failed: sstate_create_package (see /srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/build/tmp/work/core2-poky-linux/rpm/5.4.9-r59/temp/log.do_populate_sysroot.30704 for further information) NOTE: recipe rpm-5.4.9-r59: task do_populate_sysroot: Failed ERROR: Task 526 (/srv/home/pokybuild/yocto-autobuilder/yocto-slave/fri2-noemgd/build/meta/recipes-devtools/rpm/rpm_5.4.9.bb, do_populate_sysroot) failed with exit code '1' 28634 1 elizabeth.flanagan 2013-01-08 22:08:25 +0000 We've seen this before, generally during times of high network load. Reassigning to Michael Halstead to see if he has some ideas. It's a generally rare occurance. 28636 2 mhalstead 2013-01-08 22:30:18 +0000 I do not see high network load on nas02. At this point I suspect this is related to the shortcomings of NFSv3. The nas01 is using NFSv4 and as far as I know we don't see these errors there. nas02 was switched to provide NFSv3 after it was upgraded to OpenSUSE 12.2 from Ubuntu 12.04LTS and idmapd failed to work correctly. Attempts to fix idmapd failed at the time. I need to schedule 3 hours of downtime for ab03, ab07, ab08, and ab09 in order to attempt to switch back to NFSv4 after applying all available updates. 28739 3 mhalstead 2013-01-10 22:54:38 +0000 The relevant builders have been upgraded to openSUSE 12.2 and mounts have been switched to NFSv4. I expect this will fix the stale NFS handle problem.