Bug 7954

Summary: pigz has resource race issues
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: devtools / tool chainAssignee: Unassigned <unassigned>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: meta.mr.watcher, meta.watcher, mhalstead, randy.macleod, ross.burton
Version: 1.8   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Open file descriptor logging during the build none

Description Richard Purdie 2015-07-03 14:45:07 UTC 
A series of builds hung on centos6, a machine which has our 1.8 SDK installed in /opt/poky and has pigz as gzip.

e.g. https://autobuilder.yoctoproject.org/main/builders/nightly-qa-skeleton/builds/369/steps/BuildImages/logs/stdio

it would appear the machine could no longer fork (builds were running on the kernel AB and the main AB) and this resulted in various hung gzip processes:

 300 ?        S      0:00 [crypto/12]
  301 ?        S      0:00 [crypto/13]
  302 ?        S      0:00 [crypto/14]
  303 ?        S      0:00 [crypto/15]
  304 ?        S      0:00 [crypto/16]
  305 ?        S      0:00 [crypto/17]
  306 ?        S      0:00 [crypto/18]
  307 ?        S      0:00 [crypto/19]
  308 ?        S      0:00 [crypto/20]
  309 ?        S      0:00 [crypto/21]
  310 ?        S      0:00 [crypto/22]
  311 ?        S      0:00 [crypto/23]
  319 ?        S      0:00 [kthrotld/0]
  320 ?        S      0:00 [kthrotld/1]
  321 ?        S      0:00 [kthrotld/2]
  322 ?        S      0:00 [kthrotld/3]
  323 ?        S      0:00 [kthrotld/4]
  324 ?        S      0:00 [kthrotld/5]
  325 ?        S      0:00 [kthrotld/6]
  326 ?        S      0:00 [kthrotld/7]
  327 ?        S      0:00 [kthrotld/8]
  328 ?        S      0:00 [kthrotld/9]
  329 ?        S      0:00 [kthrotld/10]
  330 ?        S      0:00 [kthrotld/11]
  331 ?        S      0:00 [kthrotld/12]
  332 ?        S      0:00 [kthrotld/13]
  333 ?        S      0:00 [kthrotld/14]
  334 ?        S      0:00 [kthrotld/15]
  335 ?        S      0:00 [kthrotld/16]
  336 ?        S      0:00 [kthrotld/17]
  337 ?        S      0:00 [kthrotld/18]
  338 ?        S      0:00 [kthrotld/19]
  339 ?        S      0:00 [kthrotld/20]
  340 ?        S      0:00 [kthrotld/21]
  341 ?        S      0:00 [kthrotld/22]
  342 ?        S      0:00 [kthrotld/23]
  344 ?        S      0:00 [kpsmoused]
  345 ?        S      0:00 [usbhid_resumer]
  346 ?        S      0:00 [deferwq]
  378 ?        S      0:00 [kdmremove]
  379 ?        S      0:00 [kstriped]
  527 ?        S      0:00 [scsi_eh_0]
  563 ?        S      0:00 [scsi_eh_1]
  564 ?        S      0:00 [scsi_eh_2]
  648 ?        S      0:31 [kdmflush]
  649 ?        S      0:00 [kdmflush]
  667 ?        S      1:09 [jbd2/dm-0-8]
  668 ?        S      0:00 [ext4-dio-unwrit]
  754 ?        S<s    0:00 /sbin/udevd -d
  793 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/build/tmp/work/core2-64-poky-linux/rpm/5.4.14-r0/temp/run.sstate_create_package.12951
  843 ?        SNs    0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
  847 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build dbus-test-ptest /home/pokybuild/yocto-autobuilder-k
  927 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//21/sstate:rpm:core2-64-poky-linux:5.4.14:r0:core2-64:3:21ea08e26a0dbf93e371565aea1efeeb_package_write_rpm.tgz.PATbucE7
  948 ?        SNl    0:00 gzip
 1035 ?        S      0:00 [edac-poller]
 1556 ?        SN     0:02 [kipmi0]
 1593 ?        S      0:00 [kvm-irqfd-clean]
 1608 ?        S      0:41 [kdmflush]
 1646 ?        S      0:00 [jbd2/sda1-8]
 1647 ?        S      0:00 [ext4-dio-unwrit]
 1648 ?        S     22:42 [jbd2/dm-2-8]
 1649 ?        S      0:00 [ext4-dio-unwrit]
 1678 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build atk-locale-bg /home/pokybuild/yocto-autobuilder-ker
 1691 ?        S      0:01 [kauditd]
 1850 ?        S      1:40 [flush-253:0]
 1874 ?        SNs    0:07 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake-worker decafbad
 1930 ?        S<sl   0:06 auditd
 1950 ?        Sl     0:01 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
 1980 ?        Ss     1:48 irqbalance --pid=/var/run/irqbalance.pid
 1996 ?        Ss     0:07 rpcbind
 2016 ?        Ss     0:00 rpc.statd
 2099 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build perl-module-pod-man /home/pokybuild/yocto-autobu
 2157 ?        SN     0:00 tar -cz --format=gnu -f /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/build/tmp/work/core2-64-poky-linux/perl/5.22.0-r0/deploy-ipks/core2-64/IPKG_BUILD.3
 2201 ?        SN     0:00 gzip
 3966 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build dbus-test-ptest /home/pokybuild/yocto-autobuilder-k
 3987 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/work/core2-32-poky-linux/libproxy/0.4.11-r0/temp/run.sstate_create_package.11794
 4016 ?        SN     0:00 tar -X /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/work/core2-64-poky-linux/dbus-test/1.8.18-r0/deploy-ipks/core2-64/IPKG_BUILD.847/tarX -cz --f
 4049 ?        SNl    0:00 gzip
 4256 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//b2/sstate:libproxy:core2-32-poky-linux:0.4.11:r0:core2-32:3:b2ba1cc894aa6a38e8deafe83d6963ee_package.tgz.iI6gwuxl pack
 4298 ?        SNl    0:00 gzip
 4829 ?        Ss     0:07 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
 4861 ?        SNs    0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
 4993 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build atk-locale-bg /home/pokybuild/yocto-autobuilder-ker
 5098 ?        SN     0:00 tar -cz --format=gnu -f /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/work/core2-64-poky-linux/atk/2.16.0-r0/deploy-ipks/core2-64/IPKG_BUILD.1678/
 5137 ?        SNl    0:00 gzip
 5921 ?        S      0:00 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo bitbake-worker decafbad
 5926 ?        S      1:04 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake-worker decafbad
 5972 ?        S      0:00 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo bitbake-worker decafbad
 5999 ?        S      1:27 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/bitbake/bin/bitbake-worker decafbad
 6652 ?        Ss     0:00 dbus-daemon --system
 6684 ?        S      0:34 [rpciod/0]
 6685 ?        S      0:07 [rpciod/1]
 6686 ?        S      0:20 [rpciod/2]
 6687 ?        S      0:07 [rpciod/3]
 6688 ?        S      0:08 [rpciod/4]
 6689 ?        S      0:07 [rpciod/5]
 6690 ?        S      0:08 [rpciod/6]
 6691 ?        S      0:06 [rpciod/7]
 6692 ?        S      0:07 [rpciod/8]
 6693 ?        S      0:06 [rpciod/9]
 6694 ?        S      0:07 [rpciod/10]
 6695 ?        S      0:06 [rpciod/11]
 6696 ?        S      0:08 [rpciod/12]
 6697 ?        S      0:06 [rpciod/13]
 6698 ?        S     16:24 [rpciod/14]
 6699 ?        S      0:06 [rpciod/15]
 6700 ?        S      0:07 [rpciod/16]
 6701 ?        S      0:06 [rpciod/17]
 6702 ?        S      0:07 [rpciod/18]
 6703 ?        S      0:06 [rpciod/19]
 6704 ?        S      0:06 [rpciod/20]
 6705 ?        S      0:06 [rpciod/21]
 6706 ?        S      0:06 [rpciod/22]
 6707 ?        S      0:06 [rpciod/23]
 6709 ?        S<     0:00 [kslowd000]
 6710 ?        S<     0:00 [kslowd001]
 6711 ?        S      5:43 [nfsiod]
 6712 ?        S      0:45 [nfsv4.0-svc]
 6722 ?        Ss     0:00 /usr/sbin/acpid
 6732 ?        Ssl    0:11 hald
 6733 ?        S      0:00 hald-runner
 6776 ?        S      0:00 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0
 6777 ?        S      0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
 6797 ?        Ss     0:00 /usr/sbin/sshd
 6806 ?        Ss     0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
 6815 ?        Ss     0:31 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 6892 ?        Ss     0:06 /usr/libexec/postfix/master
 6902 ?        S      0:01 qmgr -l -t fifo -u
 6917 ?        Ss     0:00 /usr/sbin/abrtd
 6925 ?        Ss     0:00 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
 6940 ?        S      0:22 /bin/bash /usr/sbin/ksmtuned
 6951 ?        Ss     0:04 crond
 6964 ?        Ss     0:00 /usr/sbin/atd
 7777 ?        S      0:24 /usr/bin/Xvnc :1 -desktop centos6.osl.yoctoproject.org:1 (pokybuild) -auth /home/pokybuild/.Xauthority -geometry 1024x768 -rfbwait 30000 -rfbauth /home/pokybuild/.vnc/passwd -rfbport 5
 7790 tty1     Ss+    0:00 /sbin/mingetty /dev/tty1
 7792 tty2     Ss+    0:00 /sbin/mingetty /dev/tty2
 7794 tty3     Ss+    0:00 /sbin/mingetty /dev/tty3
 7797 tty4     Ss+    0:00 /sbin/mingetty /dev/tty4
 7799 tty5     Ss+    0:00 /sbin/mingetty /dev/tty5
 7801 tty6     Ss+    0:00 /sbin/mingetty /dev/tty6
 7805 ?        S<     0:00 /sbin/udevd -d
 7806 ?        S<     0:00 /sbin/udevd -d
 7812 ?        Sl     0:00 /usr/sbin/console-kit-daemon --no-daemon
 7889 ?        S      0:00 /usr/bin/xterm -geometry 80x50-50+150
 7892 pts/0    Ss+    0:00 bash
 8055 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/work/core2-64-poky-linux/valgrind/3.10.1-r1/temp/run.sstate_create_package.19387
 8275 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//72/sstate:valgrind:core2-64-poky-linux:3.10.1:r1:core2-64:3:726a9990e8698e747af1ab05912ddd13_package_write_ipk.tgz.I1y
 8325 ?        SNl    0:00 gzip
 9393 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/util-linux/2.26.2-r0/temp/run.sstate_create_package.18323
 9613 ?        Ss     0:00 sshd: halstead [priv]
 9724 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//4a/sstate:util-linux:ppc7400-poky-linux:2.26.2:r0:ppc7400:3:4ac91c5cc5af51776b833a7e8b5090cc_package_write_ipk.tgz.tOi
 9765 ?        SNl    0:00 gzip
 9808 ?        Ss     1:39 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
 9862 ?        S      0:05 sshd: halstead@pts/2
 9879 pts/2    Ss+    0:00 -bash
10060 ?        Ss     0:02 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
10677 ?        SNs    0:05 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake-worker decafbad
11085 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build perl-module-extutils-testlib /home/pokybuild/yocto-autobui
11134 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build lttng-tools-doc /home/pokybuild/yocto-autobuilder-k
11254 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/build/tmp/work/core2-64-poky-linux/rpm/5.4.14-r0/temp/run.sstate_create_package.15136
11538 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//e4/sstate:rpm:core2-64-poky-linux:5.4.14:r0:core2-64:3:e4d7e7418ee2470b94c899b8a279a51b_populate_sysroot.tgz.9gs1E9W9
11576 ?        SNl    0:00 gzip
11794 ?        SNs    0:05 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake-worker decafbad
11849 ?        Ss     0:02 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
12951 ?        SNs    0:05 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/bitbake/bin/bitbake-worker decafbad
12981 ?        Ss     0:07 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
13054 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build perl-module-extutils-testlib /home/pokybuild/yocto-autobui
13066 ?        SN     0:00 tar -cz --format=gnu -f /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/work/core2-32-poky-linux/perl/5.22.0-r0/deploy-ipks/core2-32/IPKG_BUILD.11085/contr
13067 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build lttng-tools-doc /home/pokybuild/yocto-autobuilder-k
13087 ?        SN     0:00 gzip
13096 ?        SN     0:00 tar -X /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/work/core2-64-poky-linux/lttng-tools/2.6.0-r0/deploy-ipks/core2-64/IPKG_BUILD.11134/tarX -cz
13127 ?        SNl    0:00 gzip
14410 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/build/tmp/work/core2-64-poky-linux/glib-2.0/1_2.44.1-r0/temp/run.sstate_create_package.10677
14502 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//fe/sstate:glib-2.0:core2-64-poky-linux:2.44.1:r0:core2-64:3:feeda92e608450c9fe4e848cb53cdca6_packagedata.tgz.iIGnBGVh
14519 ?        SNl    0:00 gzip
15136 ?        SNs    0:05 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/bitbake/bin/bitbake-worker decafbad
15564 ?        S      0:00 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo bitbake-worker decafbad
15569 ?        S      1:39 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake-worker decafbad
16771 ?        SNs    0:34 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/bitbake/bin/bitbake-worker decafbad
17268 ?        S      0:00 pickup -l -t fifo -u
17764 ?        S      0:00 [flush-253:2]
17777 ?        Ss     0:00 sshd: rpurdie [priv]
17779 ?        R      0:00 sshd: rpurdie@pts/4
17780 pts/4    Ss     0:00 -bash
17939 ?        S      0:00 sleep 60
17940 pts/4    R+     0:00 ps ax
18296 ?        Ss     0:07 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
18323 ?        SNs    0:06 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/bitbake/bin/bitbake-worker decafbad
18605 ?        Ss     0:03 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
18830 ?        SNs    0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
18923 ?        SNs    0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
19387 ?        SNs    0:04 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
19530 ?        SNs    0:04 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake-worker decafbad
22455 ?        Ss     0:02 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
23432 ?        S      0:00 /bin/sh -c . ./oe-init-build-env; bitbake -k core-image-kernel-dev
23506 ?        S      0:57 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake -k core-image-kerne
24237 ?        Sl     8:13 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake -k core-image-kerne
25307 ?        S      0:00 /bin/sh -c . ./oe-init-build-env; bitbake -k core-image-kernel-dev
25357 ?        S      1:02 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/bitbake/bin/bitbake -k core-image-kernel-d
25726 ?        Sl     8:29 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/bitbake/bin/bitbake -k core-image-kernel-d
26442 ?        SNs    0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
26671 ?        SNs    0:07 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake-worker decafbad
28288 ?        S      0:52 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
29082 ?        Sl     0:31 /usr/bin/python /home/pokybuild/yocto-autobuilder/bin/twistd --no_save -y buildbot.tac
29088 ?        Sl     0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 bin/buildworker-janitor /home/pokybuild/yocto-autobuilder/config/autobuilder.conf
29099 ?        SNs    0:11 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake-worker decafbad
29127 ?        SNs    0:11 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake-worker decafbad
29253 ?        S      0:59 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/bitbake/bin/bitbake-worker decafbad
29782 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build glibc-binary-localedata-es-py.iso-8859-1 /home/pokybui
29783 ?        S      0:00 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo bitbake-worker decafbad
29787 ?        S      0:46 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
29928 ?        Ss     1:54 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
29946 ?        Sl     1:03 /usr/bin/python /home/pokybuild/yocto-autobuilder-kernel/bin/twistd --no_save -y buildbot.tac
29951 ?        Sl     0:12 /usr/bin/python /home/pokybuild/yocto-autobuilder-kernel/bin/twistd --no_save -y buildbot.tac
29952 ?        Sl     0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 bin/buildworker-janitor /home/pokybuild/yocto-autobuilder-kernel/config/autobuilder.conf
30530 ?        S      0:00 /bin/sh -c . ./oe-init-build-env; bitbake -k core-image-sato
30552 ?        S      0:00 /bin/sh -c . ./oe-init-build-env; bitbake -k core-image-sato
30602 ?        S      1:09 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/bitbake/bin/bitbake -k core-image-s
30621 ?        S      1:09 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake -k core-image-sa
30656 ?        S      0:00 /bin/sh -c . ./oe-init-build-env; bitbake -k -w 'bitbake_eventlog.json' core-image-sato-sdk
30691 ?        S      1:16 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake -k -w bitbake_eventlog.jso
30716 ?        Sl    10:52 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/bitbake/bin/bitbake -k core-image-s
30773 ?        Sl    10:02 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake -k core-image-sa
30864 ?        Sl    14:07 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake -k -w bitbake_eventlog.jso
31045 ?        Ss     2:27 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
31568 ?        S      1:31 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-logrotate/build/bitbake/bin/bitbake-worker decafbad
31582 ?        SNs    0:00 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-x86-64/build/bitbake/bin/bitbake-worker decafbad
31593 ?        S      1:22 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/bitbake/bin/bitbake-worker decafbad
31618 ?        S      1:44 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/bitbake/bin/bitbake-worker decafbad
32088 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/ptest-x86/build/build/tmp/work/core2-32-poky-linux/pango/1.36.8-r0/temp/run.sstate_create_package.19530
32271 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build glibc-binary-localedata-es-py.iso-8859-1 /home/pokybui
32293 ?        SN     0:00 tar -czf /srv/www/vhosts/autobuilder.yoctoproject.org/pub/sstate//9f/sstate:pango:core2-32-poky-linux:1.36.8:r0:core2-32:3:9fdebd3ee5521db1ff4c6d4284ffc45f_populate_sysroot.tgz.V6xyayF
32305 ?        SN     0:00 tar -cz --format=gnu -f /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/glibc-locale/2.21-r0/deploy-ipks/ppc7400/IPKG_BUILD.297
32324 ?        SNl    0:00 gzip
32337 ?        SN     0:00 gzip
32613 ?        S      0:00 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo bitbake-worker decafbad
32623 ?        S      0:48 /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/python2.7 /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/bitbake/bin/bitbake-worker decafbad
32699 ?        SN     0:00 /bin/sh /home/pokybuild/yocto-autobuilder/yocto-controller/nightly-qa-skeleton/build/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-build perl-module-pod-man /home/pokybuild/yocto-autobu
[rpurdie@centos6 ~]


autom4te[13297]: segfault at 11863678 ip 00007ff5f0e9bba9 sp 00007fffe9960d90 error 4 in Fcntl.so[7ff5f0e9a000+3000]
autom4te[28191]: segfault at 15dee778 ip 00007fa27a866ba9 sp 00007fff13eb4bc0 error 4 in Fcntl.so[7fa27a865000+3000]
autom4te[15797]: segfault at 813bcf8 ip 00007f46b70e2ba9 sp 00007ffff3731520 error 4 in Fcntl.so[7f46b70e1000+3000]
autom4te[12375]: segfault at 145d9678 ip 00007f0be1c24ba9 sp 00007fff41f5e370 error 4 in Fcntl.so[7f0be1c23000+3000]
autom4te[4953]: segfault at ce86278 ip 00007f7d59407ba9 sp 00007fffea82f310 error 4 in Fcntl.so[7f7d59406000+3000]
autom4te[29942]: segfault at 10398f78 ip 00007f5cdfd20ba9 sp 00007fffe21b0160 error 4 in Fcntl.so[7f5cdfd1f000+3000]
autom4te[25783]: segfault at 7bd8a78 ip 00007f7017f57ba9 sp 00007fffd5fb6150 error 4 in Fcntl.so[7f7017f56000+3000]
autom4te[10309]: segfault at 167f36f8 ip 00007f4f314dfba9 sp 00007fffebaf39f0 error 4 in Fcntl.so[7f4f314de000+3000]
autom4te[20382]: segfault at 16459ff8 ip 00007fe4202b1ba9 sp 00007fff9d97beb0 error 4 in Fcntl.so[7fe4202b0000+3000]
autom4te[3916]: segfault at 85b6478 ip 00007fa14a71eba9 sp 00007fffcd992600 error 4 in Fcntl.so[7fa14a71d000+3000]
autom4te[3951]: segfault at 14992cf8 ip 00007f4e45190ba9 sp 00007ffff75f9630 error 4 in Fcntl.so[7f4e4518f000+3000]
autom4te[32132]: segfault at 75c0378 ip 00007f6134eacba9 sp 00007fffbe49bc90 error 4 in Fcntl.so[7f6134eab000+3000]
autom4te[1891]: segfault at fbd35f8 ip 00007f8096935ba9 sp 00007fff3b65a690 error 4 in Fcntl.so[7f8096934000+3000]
autom4te[20845]: segfault at 94d6678 ip 00007f0ae9ee7ba9 sp 00007fff7cf06750 error 4 in Fcntl.so[7f0ae9ee6000+3000]
autom4te[29955]: segfault at 14d9cef8 ip 00007f9e3cae2ba9 sp 00007fffb34427f0 error 4 in Fcntl.so[7f9e3cae1000+3000]
autom4te[6229]: segfault at a1d3978 ip 00007feaea468ba9 sp 00007fffb7d7f580 error 4 in Fcntl.so[7feaea467000+3000]
autom4te[10646]: segfault at 151d24f8 ip 00007f0dada85ba9 sp 00007fffdc397300 error 4 in Fcntl.so[7f0dada84000+3000]
autom4te[30164]: segfault at dbcd578 ip 00007fdc85461ba9 sp 00007fffc11a0830 error 4 in Fcntl.so[7fdc85460000+3000]
autom4te[718]: segfault at 14b2c578 ip 00007f9d4b44cba9 sp 00007fff9bb656d0 error 4 in Fcntl.so[7f9d4b44b000+3000]
autom4te[5500]: segfault at 83437f8 ip 00007f85eeaa4ba9 sp 00007fff33bad8f0 error 4 in Fcntl.so[7f85eeaa3000+3000]
autom4te[9684]: segfault at 12087a78 ip 00007ff175f58ba9 sp 00007fff41c61880 error 4 in Fcntl.so[7ff175f57000+3000]
autom4te[476]: segfault at acdb7f8 ip 00007f23007f6ba9 sp 00007fff38ee4fa0 error 4 in Fcntl.so[7f23007f5000+3000]
autom4te[17301]: segfault at 7626af8 ip 00007fb472576ba9 sp 00007fffbc3dc590 error 4 in Fcntl.so[7fb472575000+3000]
autom4te[26688]: segfault at 10162f78 ip 00007f8e1ece6ba9 sp 00007fff752dbef0 error 4 in Fcntl.so[7f8e1ece5000+3000]
autom4te[23457]: segfault at b6207f8 ip 00007fc3e31e9ba9 sp 00007fff8f5875f0 error 4 in Fcntl.so[7fc3e31e8000+3000]
autom4te[3300]: segfault at 589ebf8 ip 00007f038d729ba9 sp 00007fff7b3f1310 error 4 in Fcntl.so[7f038d728000+3000]
autom4te[3422]: segfault at d6a0e78 ip 00007f7f3147bba9 sp 00007fffac6bceb0 error 4 in Fcntl.so[7f7f3147a000+3000]
autom4te[2616]: segfault at dcabef8 ip 00007f1394ea4ba9 sp 00007fffde229a90 error 4 in Fcntl.so[7f1394ea3000+3000]
autom4te[21843]: segfault at 13f3e4f8 ip 00007f13dc10eba9 sp 00007fffdf68e100 error 4 in Fcntl.so[7f13dc10d000+3000]
autom4te[20412]: segfault at df0f378 ip 00007fc25fa89ba9 sp 00007fff2e6735e0 error 4 in Fcntl.so[7fc25fa88000+3000]
autom4te[26531]: segfault at 959f278 ip 00007fef3780dba9 sp 00007fff7b341a20 error 4 in Fcntl.so[7fef3780c000+3000]
autom4te[24754]: segfault at 6575e78 ip 00007fe310ca1ba9 sp 00007fff7ab6f690 error 4 in Fcntl.so[7fe310ca0000+3000]
autom4te[15776]: segfault at 16735e78 ip 00007f073e26dba9 sp 00007fff36d07f80 error 4 in Fcntl.so[7f073e26c000+3000]
autom4te[12121]: segfault at 155b6778 ip 00007feadbb08ba9 sp 00007fff84772640 error 4 in Fcntl.so[7feadbb07000+3000]
autom4te[13527]: segfault at 9a9ecf8 ip 00007fe573e4fba9 sp 00007fff1975a370 error 4 in Fcntl.so[7fe573e4e000+3000]
autom4te[15045]: segfault at daab0f8 ip 00007f3491cacba9 sp 00007fffc84a4df0 error 4 in Fcntl.so[7f3491cab000+3000]
autom4te[14069]: segfault at 158e1df8 ip 00007feabb2d7ba9 sp 00007fffd1eea0b0 error 4 in Fcntl.so[7feabb2d6000+3000]
autom4te[31905]: segfault at e709d78 ip 00007f7aa311fba9 sp 00007fff0c16d660 error 4 in Fcntl.so[7f7aa311e000+3000]
autom4te[24969]: segfault at 14894bf8 ip 00007fda72c14ba9 sp 00007fff33d7bad0 error 4 in Fcntl.so[7fda72c13000+3000]
autom4te[26206]: segfault at cc01df8 ip 00007f4d37a88ba9 sp 00007fff7ef1fab0 error 4 in Fcntl.so[7f4d37a87000+3000]
autom4te[27846]: segfault at 12924ef8 ip 00007ff96e8b2ba9 sp 00007fff94ff2460 error 4 in Fcntl.so[7ff96e8b1000+3000]
autom4te[1072]: segfault at 48eac78 ip 00007f8cc5c6aba9 sp 00007fffdfc868c0 error 4 in Fcntl.so[7f8cc5c69000+3000]
autom4te[20275]: segfault at e9c77f8 ip 00007f9d95850ba9 sp 00007fff981d4e20 error 4 in Fcntl.so[7f9d9584f000+3000]
autom4te[21051]: segfault at d713af8 ip 00007fbf4df8dba9 sp 00007fffff8b6660 error 4 in Fcntl.so[7fbf4df8c000+3000]
autom4te[20652]: segfault at d8b27f8 ip 00007f0a4c11aba9 sp 00007fff0611abb0 error 4 in Fcntl.so[7f0a4c119000+3000]
autom4te[26810]: segfault at b34e378 ip 00007fcfa4d44ba9 sp 00007fffb8d13c40 error 4 in Fcntl.so[7fcfa4d43000+3000]
autom4te[19607]: segfault at 97174f8 ip 00007f7081c51ba9 sp 00007fff21083c40 error 4 in Fcntl.so[7f7081c50000+3000]
autom4te[23852]: segfault at 51437f8 ip 00007f20fa905ba9 sp 00007fff5e2e6770 error 4 in Fcntl.so[7f20fa904000+3000]
autom4te[18147]: segfault at f7dadf8 ip 00007f7eba88aba9 sp 00007fff4db4bb50 error 4 in Fcntl.so[7f7eba889000+3000]
autom4te[27569]: segfault at 90e7cf8 ip 00007f5db0e96ba9 sp 00007fffc750b890 error 4 in Fcntl.so[7f5db0e95000+3000]
autom4te[6167]: segfault at cba6678 ip 00007fcb6dfe9ba9 sp 00007fff66dbcf90 error 4 in Fcntl.so[7fcb6dfe8000+3000]
autom4te[24585]: segfault at e444bf8 ip 00007fea88186ba9 sp 00007fff47bde4a0 error 4 in Fcntl.so[7fea88185000+3000]
autom4te[12990]: segfault at 10fa6478 ip 00007fcb7d411ba9 sp 00007fff916ca280 error 4 in Fcntl.so[7fcb7d410000+3000]
autom4te[1523]: segfault at 1308af78 ip 00007f9fcb443ba9 sp 00007fff90bd8f30 error 4 in Fcntl.so[7f9fcb442000+3000]
autom4te[21206]: segfault at e2d9378 ip 00007fe944583ba9 sp 00007fffbd666d40 error 4 in Fcntl.so[7fe944582000+3000]
autom4te[21249]: segfault at 129eaf78 ip 00007fd6b6090ba9 sp 00007fff8a351fa0 error 4 in Fcntl.so[7fd6b608f000+3000]
autom4te[8361]: segfault at 675e578 ip 00007f1476054ba9 sp 00007fff3aed0fc0 error 4 in Fcntl.so[7f1476053000+3000]
autom4te[30555]: segfault at 91a8a78 ip 00007f6bb717cba9 sp 00007fffc9e2d9c0 error 4 in Fcntl.so[7f6bb717b000+3000]
autom4te[3669]: segfault at d0068f8 ip 00007f92a1956ba9 sp 00007ffff69bf920 error 4 in Fcntl.so[7f92a1955000+3000]
autom4te[10692]: segfault at 1016c778 ip 00007fc1e4f64ba9 sp 00007fff2a053aa0 error 4 in Fcntl.so[7fc1e4f63000+3000]
autom4te[28648]: segfault at 1476b5f8 ip 00007fd2d2879ba9 sp 00007fff21283670 error 4 in Fcntl.so[7fd2d2878000+3000]
autom4te[13585]: segfault at 14d0a378 ip 00007fea98983ba9 sp 00007fffea11b7a0 error 4 in Fcntl.so[7fea98982000+3000]
autom4te[25632]: segfault at 161c0678 ip 00007fd3746f0ba9 sp 00007fff8493e010 error 4 in Fcntl.so[7fd3746ef000+3000]
autom4te[22075]: segfault at ad4a178 ip 00007ffd3f9a0ba9 sp 00007fffcd19c540 error 4 in Fcntl.so[7ffd3f99f000+3000]
autom4te[28603]: segfault at 155dd3f8 ip 00007f3087dcaba9 sp 00007fffec5fc370 error 4 in Fcntl.so[7f3087dc9000+3000]
autom4te[13257]: segfault at 96f04f8 ip 00007f9ca6891ba9 sp 00007fff7c5f2430 error 4 in Fcntl.so[7f9ca6890000+3000]
autom4te[15849]: segfault at 85d4078 ip 00007f8caa7c3ba9 sp 00007fff20a01850 error 4 in Fcntl.so[7f8caa7c2000+3000]
autom4te[16489]: segfault at 6ad60f8 ip 00007f9a67d20ba9 sp 00007fffc8779340 error 4 in Fcntl.so[7f9a67d1f000+3000]
autom4te[20179]: segfault at a099978 ip 00007fc24f123ba9 sp 00007fffc3cedaf0 error 4 in Fcntl.so[7fc24f122000+3000]
autom4te[16578]: segfault at d731ef8 ip 00007fe6db29bba9 sp 00007fff99c50920 error 4 in Fcntl.so[7fe6db29a000+3000]
autom4te[25810]: segfault at 74524f8 ip 00007f6a04749ba9 sp 00007fff86f63bf0 error 4 in Fcntl.so[7f6a04748000+3000]
autom4te[13673]: segfault at f60c478 ip 00007f167a93aba9 sp 00007fff218bae00 error 4 in Fcntl.so[7f167a939000+3000]
autom4te[12960]: segfault at 148624f8 ip 00007f7f9521eba9 sp 00007fffd0cd0010 error 4 in Fcntl.so[7f7f9521d000+3000]
autom4te[25213]: segfault at c5b5df8 ip 00007f76cbd3fba9 sp 00007fff0303d030 error 4 in Fcntl.so[7f76cbd3e000+3000]
autom4te[16940]: segfault at 8637a78 ip 00007f9804678ba9 sp 00007fff69c7efb0 error 4 in Fcntl.so[7f9804677000+3000]
autom4te[1003]: segfault at fd9d378 ip 00007fd9117c1ba9 sp 00007fffc1fcd400 error 4 in Fcntl.so[7fd9117c0000+3000]
autom4te[7303]: segfault at 1142a478 ip 00007f7d9c519ba9 sp 00007fff59e59670 error 4 in Fcntl.so[7f7d9c518000+3000]
autom4te[21681]: segfault at 72df4f8 ip 00007f20b99efba9 sp 00007fff01d153c0 error 4 in Fcntl.so[7f20b99ee000+3000]
autom4te[4442]: segfault at b382ef8 ip 00007f5880aa3ba9 sp 00007fff2ce0f630 error 4 in Fcntl.so[7f5880aa2000+3000]
autom4te[5366]: segfault at ce49cf8 ip 00007f2b85233ba9 sp 00007fffab565750 error 4 in Fcntl.so[7f2b85232000+3000]
autom4te[7434]: segfault at 970e178 ip 00007fd19c72fba9 sp 00007fff00fcd210 error 4 in Fcntl.so[7fd19c72e000+3000]
autom4te[11649]: segfault at de3e8f8 ip 00007f46d8f6cba9 sp 00007fffe6c2a2a0 error 4 in Fcntl.so[7f46d8f6b000+3000]
autom4te[8404]: segfault at 14c882f8 ip 00007f6b85ff3ba9 sp 00007fffbdd37370 error 4 in Fcntl.so[7f6b85ff2000+3000]
autom4te[18104]: segfault at 6b6d1f8 ip 00007f7f6273bba9 sp 00007fff1ee8d200 error 4 in Fcntl.so[7f7f6273a000+3000]
autom4te[30582]: segfault at ab71978 ip 00007fe34e75dba9 sp 00007fff10e7f390 error 4 in Fcntl.so[7fe34e75c000+3000]
autom4te[17814]: segfault at e29a4f8 ip 00007f019f801ba9 sp 00007fff3f6f77e0 error 4 in Fcntl.so[7f019f800000+3000]
autom4te[19119]: segfault at a778e78 ip 00007f86eaea6ba9 sp 00007fff1636c080 error 4 in Fcntl.so[7f86eaea5000+3000]
autom4te[31558]: segfault at e0f4c78 ip 00007fcf2ce77ba9 sp 00007fff5c7336f0 error 4 in Fcntl.so[7fcf2ce76000+3000]
autom4te[32001]: segfault at 10f653f8 ip 00007f0725f69ba9 sp 00007fff9fb451d0 error 4 in Fcntl.so[7f0725f68000+3000]
autom4te[8577]: segfault at 15e3b5f8 ip 00007f941b80bba9 sp 00007fff4b8d1870 error 4 in Fcntl.so[7f941b80a000+3000]
autom4te[18146]: segfault at 7575df8 ip 00007f9171acdba9 sp 00007fff3d3384e0 error 4 in Fcntl.so[7f9171acc000+3000]
autom4te[32544]: segfault at dc3e078 ip 00007f1da98e5ba9 sp 00007fffa9d1ff20 error 4 in Fcntl.so[7f1da98e4000+3000]
autom4te[10423]: segfault at 64bed78 ip 00007fcd8c453ba9 sp 00007fff7718c1e0 error 4 in Fcntl.so[7fcd8c452000+3000]
autom4te[23543]: segfault at c85c0f8 ip 00007f3b2d30fba9 sp 00007fffbc0f7990 error 4 in Fcntl.so[7f3b2d30e000+3000]
autom4te[17116]: segfault at 4cd7578 ip 00007faea57e8ba9 sp 00007fff852098d0 error 4 in Fcntl.so[7faea57e7000+3000]
autom4te[25707]: segfault at d4084f8 ip 00007ff054c27ba9 sp 00007fffeae3f4d0 error 4 in Fcntl.so[7ff054c26000+3000]
autom4te[12745]: segfault at 77c2678 ip 00007fea2bc6cba9 sp 00007fffb2afc420 error 4 in Fcntl.so[7fea2bc6b000+3000]
autom4te[18948]: segfault at 5464f78 ip 00007fdac1d3fba9 sp 00007fff00ec21b0 error 4 in Fcntl.so[7fdac1d3e000+3000]
autom4te[23631]: segfault at 65b1378 ip 00007f69d1f8dba9 sp 00007fff259cdea0 error 4 in Fcntl.so[7f69d1f8c000+3000]
autom4te[11520]: segfault at a15af78 ip 00007f890c8ffba9 sp 00007fff134e1ea0 error 4 in Fcntl.so[7f890c8fe000+3000]
autom4te[1032]: segfault at 7fec7f8 ip 00007f600a806ba9 sp 00007fffde403100 error 4 in Fcntl.so[7f600a805000+3000]
autom4te[13615]: segfault at db985f8 ip 00007f08e051bba9 sp 00007ffff24cccd0 error 4 in Fcntl.so[7f08e051a000+3000]
autom4te[3877]: segfault at 8563ef8 ip 00007f8e49049ba9 sp 00007fff81dd4810 error 4 in Fcntl.so[7f8e49048000+3000]
Clock: inserting leap second 23:59:60 UTC
autom4te[13290]: segfault at dc98d78 ip 00007f7366095ba9 sp 00007fff790dec10 error 4 in Fcntl.so[7f7366094000+3000]
autom4te[32272]: segfault at cf61df8 ip 00007f5c076afba9 sp 00007fff13b88360 error 4 in Fcntl.so[7f5c076ae000+3000]
autom4te[19243]: segfault at 136cb4f8 ip 00007f4bce6edba9 sp 00007fff1ca23a20 error 4 in Fcntl.so[7f4bce6ec000+3000]
autom4te[31622]: segfault at 16617af8 ip 00007f86f7a0eba9 sp 00007fffdd7eb020 error 4 in Fcntl.so[7f86f7a0d000+3000]
autom4te[11569]: segfault at 8cf7178 ip 00007f362f7adba9 sp 00007fff464fe400 error 4 in Fcntl.so[7f362f7ac000+3000]
autom4te[27216]: segfault at 82651f8 ip 00007f84040a9ba9 sp 00007fff62d245a0 error 4 in Fcntl.so[7f84040a8000+3000]
autom4te[12684]: segfault at 8e9e978 ip 00007fe422a21ba9 sp 00007fff87900930 error 4 in Fcntl.so[7fe422a20000+3000]
autom4te[10820]: segfault at 85cc3f8 ip 00007f0c299b7ba9 sp 00007fffa620a960 error 4 in Fcntl.so[7f0c299b6000+3000]
autom4te[26939]: segfault at 5d27a78 ip 00007fdb4ab44ba9 sp 00007fff40f31540 error 4 in Fcntl.so[7fdb4ab43000+3000]
qemu-system-x86[18519]: segfault at 2d0 ip 00007fa132aeb880 sp 00007fa13123cae8 error 4 in libpthread-2.12.so[7fa132adf000+17000]
autom4te[31682]: segfault at d61adf8 ip 00007f5f6fe79ba9 sp 00007fff08ffdc90 error 4 in Fcntl.so[7f5f6fe78000+3000]
autom4te[745]: segfault at 12475478 ip 00007fbb2a334ba9 sp 00007fff98bfa420 error 4 in Fcntl.so[7fbb2a333000+3000]
autom4te[19854]: segfault at d13e0f8 ip 00007fef6a6f6ba9 sp 00007fffca62d080 error 4 in Fcntl.so[7fef6a6f5000+3000]
autom4te[23200]: segfault at 9ade1f8 ip 00007f16b1657ba9 sp 00007fff49f2ea20 error 4 in Fcntl.so[7f16b1656000+3000]
autom4te[4332]: segfault at 845d3f8 ip 00007f97a38ebba9 sp 00007fff56134ab0 error 4 in Fcntl.so[7f97a38ea000+3000]
INFO: task syslinux:6843 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000000     0  6843   6761 0x00000080
 ffff880197507d38 0000000000000086 ffff88084c4d5958 ffff880197507da8
 ffff880197507ca8 ffffffff8128c539 00000000000158c0 ffff88084c4ced00
 ffff880197507cf8 ffffffff8105c5d5 ffff880102231ad8 ffff880197507fd8
Call Trace:
 [<ffffffff8128c539>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8105c5d5>] ? select_idle_sibling+0x95/0x150
 [<ffffffff8152a925>] schedule_timeout+0x215/0x2e0
 [<ffffffff8105def4>] ? check_preempt_wakeup+0x1a4/0x260
 [<ffffffff8106d5eb>] ? enqueue_task_fair+0xfb/0x100
 [<ffffffff8152a5a3>] wait_for_common+0x123/0x180
 [<ffffffff81064be0>] ? default_wake_function+0x0/0x20
 [<ffffffff8152a6bd>] wait_for_completion+0x1d/0x20
 [<ffffffff811ba4b5>] writeback_inodes_sb_nr+0x75/0xa0
 [<ffffffff811ba523>] writeback_inodes_sb+0x43/0x50
 [<ffffffff811c0cde>] __sync_filesystem+0x3e/0x90
 [<ffffffff811c0e28>] sync_filesystems+0xf8/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:10200 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000004     0 10200   9474 0x00000080
 ffff880864cc7ea8 0000000000000086 0000000000000000 0000000000000007
 ffff880864cc7e28 ffffffff8106d554 00020c83d8d86563 ffff88102366a040
 ffff8800282758c0 00000001225af4fe ffff88091803d068 ffff880864cc7fd8
Call Trace:
 [<ffffffff8106d554>] ? enqueue_task_fair+0x64/0x100
 [<ffffffff8152b3a6>] __mutex_lock_slowpath+0x96/0x210
 [<ffffffff8152aecb>] mutex_lock+0x2b/0x50
 [<ffffffff811c0d56>] sync_filesystems+0x26/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:9758 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000000     0  9758   9687 0x00000080
 ffff88083005fea8 0000000000000086 0000000000000000 0000000000000015
 ffff88083005fe28 ffffffff8106d554 00020c9e27376e25 ffff88102366a040
 ffff8800283558c0 00000001225cae49 ffff88102366bad8 ffff88083005ffd8
Call Trace:
 [<ffffffff8106d554>] ? enqueue_task_fair+0x64/0x100
 [<ffffffff8152b3a6>] __mutex_lock_slowpath+0x96/0x210
 [<ffffffff81064c35>] ? wake_up_process+0x15/0x20
 [<ffffffff8152aecb>] mutex_lock+0x2b/0x50
 [<ffffffff811c0d56>] sync_filesystems+0x26/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:6843 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 000000000000000c     0  6843   6761 0x00000080
 ffff880197507d38 0000000000000086 ffffffff811244d0 ffff880197507d38
 ffff880197507ca8 ffffffff8128c539 00000000000158c0 ffff88084c40ed00
 ffff880197507cf8 ffffffff8105c5d5 ffff880102231ad8 ffff880197507fd8
Call Trace:
 [<ffffffff811244d0>] ? sync_page+0x0/0x50
 [<ffffffff8128c539>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8105c5d5>] ? select_idle_sibling+0x95/0x150
 [<ffffffff8152a925>] schedule_timeout+0x215/0x2e0
 [<ffffffff81058678>] ? resched_task+0x68/0x80
 [<ffffffff8105df12>] ? check_preempt_wakeup+0x1c2/0x260
 [<ffffffff8106d5eb>] ? enqueue_task_fair+0xfb/0x100
 [<ffffffff8152a5a3>] wait_for_common+0x123/0x180
 [<ffffffff81064be0>] ? default_wake_function+0x0/0x20
 [<ffffffff8152a6bd>] wait_for_completion+0x1d/0x20
 [<ffffffff811ba5b8>] sync_inodes_sb+0x88/0x190
 [<ffffffff811c0d22>] __sync_filesystem+0x82/0x90
 [<ffffffff811c0e28>] sync_filesystems+0xf8/0x130
 [<ffffffff811c0ec1>] sys_sync+0x21/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:10200 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000006     0 10200   9474 0x00000080
 ffff880864cc7ea8 0000000000000086 0000000000000000 0000000000000007
 ffff880864cc7e28 ffffffff8106d554 00020ca9e5aada55 ffff88102366a040
 ffff8800282758c0 00000001225d7359 ffff88091803d068 ffff880864cc7fd8
Call Trace:
 [<ffffffff8106d554>] ? enqueue_task_fair+0x64/0x100
 [<ffffffff8152b3a6>] __mutex_lock_slowpath+0x96/0x210
 [<ffffffff8152aecb>] mutex_lock+0x2b/0x50
 [<ffffffff811c0d56>] sync_filesystems+0x26/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:9758 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000000     0  9758   9687 0x00000080
 ffff88083005fea8 0000000000000086 0000000000000000 0000000000000015
 ffff88083005fe28 ffffffff8106d554 00020c9e27376e25 ffff88102366a040
 ffff8800283558c0 00000001225cae49 ffff88102366bad8 ffff88083005ffd8
Call Trace:
 [<ffffffff8106d554>] ? enqueue_task_fair+0x64/0x100
 [<ffffffff8152b3a6>] __mutex_lock_slowpath+0x96/0x210
 [<ffffffff81064c35>] ? wake_up_process+0x15/0x20
 [<ffffffff8152aecb>] mutex_lock+0x2b/0x50
 [<ffffffff811c0d56>] sync_filesystems+0x26/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:6843 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 000000000000000c     0  6843   6761 0x00000080
 ffff880197507d38 0000000000000086 ffffffff811244d0 ffff880197507d38
 ffff880197507ca8 ffffffff8128c539 00000000000158c0 ffff88084c40ed00
 ffff880197507cf8 ffffffff8105c5d5 ffff880102231ad8 ffff880197507fd8
Call Trace:
 [<ffffffff811244d0>] ? sync_page+0x0/0x50
 [<ffffffff8128c539>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8105c5d5>] ? select_idle_sibling+0x95/0x150
 [<ffffffff8152a925>] schedule_timeout+0x215/0x2e0
 [<ffffffff81058678>] ? resched_task+0x68/0x80
 [<ffffffff8105df12>] ? check_preempt_wakeup+0x1c2/0x260
 [<ffffffff8106d5eb>] ? enqueue_task_fair+0xfb/0x100
 [<ffffffff8152a5a3>] wait_for_common+0x123/0x180
 [<ffffffff81064be0>] ? default_wake_function+0x0/0x20
 [<ffffffff8152a6bd>] wait_for_completion+0x1d/0x20
 [<ffffffff811ba5b8>] sync_inodes_sb+0x88/0x190
 [<ffffffff811c0d22>] __sync_filesystem+0x82/0x90
 [<ffffffff811c0e28>] sync_filesystems+0xf8/0x130
 [<ffffffff811c0ec1>] sys_sync+0x21/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:10200 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000006     0 10200   9474 0x00000080
 ffff880864cc7ea8 0000000000000086 0000000000000000 0000000000000007
 ffff880864cc7e28 ffffffff8106d554 00020ca9e5aada55 ffff88102366a040
 ffff8800282758c0 00000001225d7359 ffff88091803d068 ffff880864cc7fd8
Call Trace:
 [<ffffffff8106d554>] ? enqueue_task_fair+0x64/0x100
 [<ffffffff8152b3a6>] __mutex_lock_slowpath+0x96/0x210
 [<ffffffff8152aecb>] mutex_lock+0x2b/0x50
 [<ffffffff811c0d56>] sync_filesystems+0x26/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:9758 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 0000000000000000     0  9758   9687 0x00000080
 ffff88083005fea8 0000000000000086 0000000000000000 0000000000000015
 ffff88083005fe28 ffffffff8106d554 00020c9e27376e25 ffff88102366a040
 ffff8800283558c0 00000001225cae49 ffff88102366bad8 ffff88083005ffd8
Call Trace:
 [<ffffffff8106d554>] ? enqueue_task_fair+0x64/0x100
 [<ffffffff8152b3a6>] __mutex_lock_slowpath+0x96/0x210
 [<ffffffff81064c35>] ? wake_up_process+0x15/0x20
 [<ffffffff8152aecb>] mutex_lock+0x2b/0x50
 [<ffffffff811c0d56>] sync_filesystems+0x26/0x130
 [<ffffffff811c0eb7>] sys_sync+0x17/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task syslinux:6843 blocked for more than 120 seconds.
      Not tainted 2.6.32-504.23.4.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syslinux      D 000000000000000c     0  6843   6761 0x00000080
 ffff880197507d38 0000000000000086 ffffffff811244d0 ffff880197507d38
 ffff880197507ca8 ffffffff8128c539 00000000000158c0 ffff88084c40ed00
 ffff880197507cf8 ffffffff8105c5d5 ffff880102231ad8 ffff880197507fd8
Call Trace:
 [<ffffffff811244d0>] ? sync_page+0x0/0x50
 [<ffffffff8128c539>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8105c5d5>] ? select_idle_sibling+0x95/0x150
 [<ffffffff8152a925>] schedule_timeout+0x215/0x2e0
 [<ffffffff81058678>] ? resched_task+0x68/0x80
 [<ffffffff8105df12>] ? check_preempt_wakeup+0x1c2/0x260
 [<ffffffff8106d5eb>] ? enqueue_task_fair+0xfb/0x100
 [<ffffffff8152a5a3>] wait_for_common+0x123/0x180
 [<ffffffff81064be0>] ? default_wake_function+0x0/0x20
 [<ffffffff8152a6bd>] wait_for_completion+0x1d/0x20
 [<ffffffff811ba5b8>] sync_inodes_sb+0x88/0x190
 [<ffffffff811c0d22>] __sync_filesystem+0x82/0x90
 [<ffffffff811c0e28>] sync_filesystems+0xf8/0x130
 [<ffffffff811c0ec1>] sys_sync+0x21/0x40
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
autom4te[14195]: segfault at d59f4f8 ip 00007fb0b8caaba9 sp 00007fff81177170 error 4 in Fcntl.so[7fb0b8ca9000+3000]
autom4te[21463]: segfault at 1330e1f8 ip 00007f750c570ba9 sp 00007fff7ca83770 error 4 in Fcntl.so[7f750c56f000+3000]
autom4te[24400]: segfault at ffc0d78 ip 00007f06bc18dba9 sp 00007fffe21f6c10 error 4 in Fcntl.so[7f06bc18c000+3000]
autom4te[31112]: segfault at 8d83bf8 ip 00007f4aa9fd6ba9 sp 00007fff5f66b160 error 4 in Fcntl.so[7f4aa9fd5000+3000]
autom4te[21950]: segfault at a246df8 ip 00007ff6e9869ba9 sp 00007fff59968f90 error 4 in Fcntl.so[7ff6e9868000+3000]



[rpurdie@centos6 ~]$ pstree
init─┬─Xvnc
     ├─abrt-dump-oops
     ├─abrtd
     ├─acpid
     ├─atd
     ├─auditd───{auditd}
     ├─console-kit-dae───63*[{console-kit-da}]
     ├─crond
     ├─dbus-daemon
     ├─hald─┬─hald-runner─┬─hald-addon-acpi
     │      │             └─hald-addon-inpu
     │      └─{hald}
     ├─irqbalance
     ├─ksmtuned───sleep
     ├─master─┬─pickup
     │        └─qmgr
     ├─6*[mingetty]
     ├─ntpd
     ├─10*[pseudo]
     ├─2*[python2.7.real───2*[{python2.7.real}]]
     ├─rpc.statd
     ├─rpcbind
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd─┬─sshd───sshd───bash
     │      └─sshd───sshd───bash───pstree
     ├─twistd─┬─sh───python2.7.real───python2.7.real─┬─pseudo───python2.7.real───python2.7.real───run.sstate_crea───tar───gzip───16*[{gzip}]
     │        │                                      ├─python2.7.real───python2.7.real───run.sstate_crea───tar───gzip───14*[{gzip}]
     │        │                                      └─3*[{python2.7.real}]
     │        ├─sh───python2.7.real───python2.7.real─┬─pseudo───python2.7.real─┬─python2.7.real───opkg-build───opkg-build───tar───gzip
     │        │                                      │                         └─python2.7.real
     │        │                                      ├─python2.7.real───python2.7.real───run.sstate_crea───tar───gzip───2*[{gzip}]
     │        │                                      └─3*[{python2.7.real}]
     │        ├─sh───python2.7.real───python2.7.real─┬─pseudo───python2.7.real─┬─python2.7.real───opkg-build───opkg-build───tar───gzip
     │        │                                      │                         ├─python2.7.real───run.sstate_crea───tar───gzip───13*[{gzip}]
     │        │                                      │                         └─python2.7.real
     │        │                                      ├─python2.7.real───python2.7.real───run.sstate_crea───tar───gzip───7*[{gzip}]
     │        │                                      └─3*[{python2.7.real}]
     │        └─{twistd}
     ├─twistd───17*[{twistd}]
     ├─twistd─┬─sh───python2.7.real───python2.7.real─┬─pseudo───python2.7.real─┬─3*[python2.7.real]
     │        │                                      │                         ├─python2.7.real───opkg-build───opkg-build───tar───gzip───{gzip}
     │        │                                      │                         ├─python2.7.real───run.sstate_crea───tar───gzip───23*[{gzip}]
     │        │                                      │                         ├─python2.7.real───opkg-build───opkg-build───tar───gzip───2*[{gzip}]
     │        │                                      │                         └─python2.7.real───opkg-build───opkg-build───tar───gzip───11*[{gzip}]
     │        │                                      ├─python2.7.real
     │        │                                      └─3*[{python2.7.real}]
     │        ├─sh───python2.7.real───python2.7.real─┬─pseudo───python2.7.real─┬─python2.7.real───opkg-build───opkg-build───tar───gzip
     │        │                                      │                         └─python2.7.real───run.sstate_crea───tar───gzip───17*[{gzip}]
     │        │                                      ├─python2.7.real
     │        │                                      └─3*[{python2.7.real}]
     │        └─{twistd}
     ├─udevd───2*[udevd]
     ├─xinetd
     └─xterm───bash

The gzip processes were in futexes:

rpurdie]# gdb -p 32305
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 32305
Reading symbols from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/tar...Missing separate debuginfo for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/tar
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3a/f0af0ee8fdc1c7785821093610e17d1faa3d56.debug
(no debugging symbols found)...done.
Reading symbols from /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/../lib/pseudo/lib64/libpseudo.so...(no debugging symbols found)...done.
Loaded symbols for /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr/bin/../lib/pseudo/lib64/libpseudo.so
Reading symbols from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libc.so.6...Missing separate debuginfo for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libc.so.6
(no debugging symbols found)...done.
Loaded symbols for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libc.so.6
Reading symbols from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libdl.so.2...Missing separate debuginfo for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libdl.so.2
(no debugging symbols found)...done.
Loaded symbols for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libdl.so.2
Reading symbols from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libpthread.so.0...Missing separate debuginfo for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libpthread.so.0
(no debugging symbols found)...done.

warning: File "/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libthread_db.so.1
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libpthread.so.0
Reading symbols from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2...Missing separate debuginfo for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2
(no debugging symbols found)...done.

warning: File "/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2

warning: File "/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
0x00007f82f5444b8a in waitpid () from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libc.so.6
(gdb) bt
#0  0x00007f82f5444b8a in waitpid () from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libc.so.6
#1  0x0000000000421388 in ?? ()
#2  0x0000000000407cc3 in ?? ()
#3  0x000000000040dc52 in ?? ()
#4  0x000000000040607c in ?? ()
#5  0x00007f82f53ad810 in __libc_start_main () from /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/../../lib/libc.so.6
#6  0x0000000000406429 in ?? ()
(gdb) q
A debugging session is active.

	Inferior 1 [process 32305] will be detached.

Quit anyway? (y or n) y
Detaching from program: /opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/tar, process 32305
[root@centos6 rpurdie]# ls /proc/32337/fd/
0  1  10  11  12  13  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34  35  36  37  38  39  4  40  41  43  5  6  7  8  9
[root@centos6 rpurdie]# ls /proc/32337/fd/ -la
total 0
dr-x------ 2 pokybuild users  0 Jul  3 13:21 .
dr-xr-xr-x 8 pokybuild users  0 Jul  3 09:15 ..
lr-x------ 1 pokybuild users 64 Jul  3 13:21 0 -> pipe:[505258883]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 1 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/glibc-locale/2.21-r0/deploy-ipks/ppc7400/IPKG_BUILD.29782/control.tar.gz
lr-x------ 1 pokybuild users 64 Jul  3 13:21 10 -> inotify
lr-x------ 1 pokybuild users 64 Jul  3 13:21 11 -> pipe:[488437917]
lrwx------ 1 pokybuild users 64 Jul  3 13:21 12 -> socket:[488436209]
lrwx------ 1 pokybuild users 64 Jul  3 13:21 13 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/bitbake.lock
lr-x------ 1 pokybuild users 64 Jul  3 13:21 14 -> /dev/null
l-wx------ 1 pokybuild users 64 Jul  3 13:21 15 -> pipe:[488437917]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 16 -> pipe:[488437921]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 17 -> pipe:[488437921]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 18 -> pipe:[488437925]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 19 -> pipe:[488437925]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 2 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/glibc-locale/2.21-r0/temp/log.do_package_write_ipk.16771
lr-x------ 1 pokybuild users 64 Jul  3 13:21 20 -> pipe:[488437929]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 21 -> pipe:[488437929]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 22 -> pipe:[497248080]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 23 -> pipe:[497205519]
lrwx------ 1 pokybuild users 64 Jul  3 13:21 24 -> /dev/null
lr-x------ 1 pokybuild users 64 Jul  3 13:21 25 -> pipe:[497234732]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 26 -> pipe:[497300527]
lrwx------ 1 pokybuild users 64 Jul  3 13:21 27 -> /dev/null
lr-x------ 1 pokybuild users 64 Jul  3 13:21 28 -> /dev/urandom
lr-x------ 1 pokybuild users 64 Jul  3 13:21 29 -> pipe:[497110097]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 3 -> pipe:[488436203]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 30 -> pipe:[497319633]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 31 -> pipe:[497278476]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 32 -> pipe:[497322874]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 33 -> pipe:[497327234]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 34 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr
lr-x------ 1 pokybuild users 64 Jul  3 13:21 35 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/glibc-locale/2.21-r0/pseudo
lrwx------ 1 pokybuild users 64 Jul  3 13:21 36 -> socket:[505258971]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 37 -> /dev/null
l-wx------ 1 pokybuild users 64 Jul  3 13:21 38 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/glibc-locale/2.21-r0/temp/log.do_package_write_ipk.16771
lrwx------ 1 pokybuild users 64 Jul  3 13:21 39 -> /dev/null
l-wx------ 1 pokybuild users 64 Jul  3 13:21 4 -> pipe:[488436203]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 40 -> pipe:[488469385]
l-wx------ 1 pokybuild users 64 Jul  3 13:21 41 -> pipe:[488435461]
lrwx------ 1 pokybuild users 64 Jul  3 13:21 43 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/glibc-locale/2.21-r0/packages-split/glibc-binary-localedata-es-py.iso-8859-1.lock
lrwx------ 1 pokybuild users 64 Jul  3 13:21 5 -> socket:[488436201]
lrwx------ 1 pokybuild users 64 Jul  3 13:21 6 -> socket:[488436202]
lr-x------ 1 pokybuild users 64 Jul  3 13:21 7 -> inotify
lrwx------ 1 pokybuild users 64 Jul  3 13:21 8 -> /dev/null
lr-x------ 1 pokybuild users 64 Jul  3 13:21 9 -> /dev/urandom
[root@centos6 rpurdie]# ls /proc/9765/fd/ -la
total 0
dr-x------ 2 pokybuild users  0 Jul  3 13:14 .
dr-xr-xr-x 8 pokybuild users  0 Jul  3 09:17 ..
lr-x------ 1 pokybuild users 64 Jul  3 13:14 0 -> pipe:[505273133]
l-wx------ 1 pokybuild users 64 Jul  3 13:14 1 -> /srv/www/vhosts/sstate.yoctoproject.org/4a/sstate:util-linux:ppc7400-poky-linux:2.26.2:r0:ppc7400:3:4ac91c5cc5af51776b833a7e8b5090cc_package_write_ipk.tgz.tOiEAvel
l-wx------ 1 pokybuild users 64 Jul  3 13:14 2 -> pipe:[505272517]
lr-x------ 1 pokybuild users 64 Jul  3 13:14 20 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/sysroots/x86_64-linux/usr
lr-x------ 1 pokybuild users 64 Jul  3 13:14 21 -> /home/pokybuild/yocto-autobuilder-kernel/yocto-worker/kernel-ppc/build/build/tmp/work/ppc7400-poky-linux/util-linux/2.26.2-r0/pseudo
lrwx------ 1 pokybuild users 64 Jul  3 13:14 22 -> socket:[505273258]


 strace -p 9765 -f
Process 9765 attached with 18 threads - interrupt to quit
[pid  9856] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9859] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9853] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9858] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9851] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9855] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9849] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9847] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9844] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9843] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9850] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9842] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9857] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9838] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9836] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9833] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9765] futex(0x63b8c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  9829] futex(0x74a75c, FUTEX_WAIT_PRIVATE, 1, NULL^C <unfinished ...>



Likely we need to look at the error paths in pigz and figure out whether failures are well handled.
Comment 1 Michael Halstead 2015-07-06 17:14:16 UTC 
Created attachment 2576 [details]
Open file descriptor logging during the build
Comment 2 Michael Halstead 2015-07-06 17:14:56 UTC 
If it's relevant it looks like pokybuild was out of allowed open file handles. 

We were nowhere near the system limit:
sysctl fs.file-nr
fs.file-nr = 1248	0	6553322

But pokybuild is much more constrained. 
ulimit -Sn
1024
ulimit -Hn
4096

During the build there were some points that the open descriptors went over 8000.

Linux 2.6.32-504.23.4.el6.x86_64 (centos6.osl.yoctoproject.org) 	07/03/2015 	_x86_64_	(24 CPU)

12:00:01 AM dentunusd   file-nr  inode-nr    pty-nr
12:10:01 AM   1301816      8064    844409         4
12:20:01 AM   1254938      6336    742260         4
12:30:02 AM   1907781      5712   1061746         4
12:40:01 AM   1657203      6624    990812         4
12:50:01 AM   2052072      7728   1160877         4
01:00:01 AM   2152702      8736   1226978         4
01:10:01 AM   2204908      8064   1227499         4
01:20:02 AM   2125034      6528   1139800         4
01:30:01 AM   2223994      8688   1174059         4
01:40:02 AM   1808354      5856    966341         4
01:50:01 AM   1853701      7008    900623         4

Should pokybuild's limits be raised?
Comment 3 Ross Burton 2015-07-06 20:04:57 UTC 
8000 does seem a little excessive but we are running multiple builds, and each bitbake will open inotify monitors for all of the layers, so that's a lot of FDs.  I say increase it.
Comment 4 Michael Halstead 2015-07-06 20:29:22 UTC 
Every builder was set to 1024/4096 except for debian8 which is 65536/65536.

Inotify watches are max 8192 on all hosts except opensuse131 where it's 65536.

I'll raise it to 65536 in every case since we have plenty of memory.

Do we need to document that for end users?
Comment 5 Ross Burton 2015-07-09 14:47:30 UTC 
To test the hypothesis that this is resource limits you can use ulimit in a shell.  -n will set the number of open file descriptors, for a start (and despite the documentation, works on Linux).
Comment 6 Randy MacLeod 2020-01-23 16:10:47 UTC 
This hasn't been seen on the YP autobuilder or by other users so closing.