Bug 12074

Summary: oe-selftest hangs in vim on debian8.yocto.io
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: devtools / tool chainAssignee: Paul Eggleton <bluelightning>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: meta.mr.watcher, meta.watcher
Version: 2.2   
Target Milestone: 2.4 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2017-09-12 15:45:24 UTC
I believe in test_devtool_add (devtool.DevtoolTests) the autobuilder locked up in vim with an ssh-agent prompt. The processes active were:

rpurdie@debian8:~$ ps ax | grep -v kworker
  PID TTY      STAT   TIME COMMAND
[...]
 1699 ?        Ss     0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
 1749 ?        Ssl    0:00 /usr/sbin/rsyslogd -n
 1753 ?        Ss     0:00 /usr/sbin/acpid
 1763 ?        Ss     3:12 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
 2560 ?        S      0:02 Xtightvnc :1 -desktop X -auth /home/pokybuild/.Xauthority -geometry 1024x768 -depth 24 -rfbwait 120000 -rfbauth /home/pokybuild/.vnc/passwd -rfbport 5901 -fp /usr/share/fonts/X11/misc/,/usr/share/fonts/X11/Type1/,/usr/share/fonts/X11/75dpi/,/usr/share/fonts/X11/100dpi/ -co /etc/X11/rgb
 2576 ?        S      0:00 /bin/sh /home/pokybuild/.vnc/xstartup
 2578 tty1     Ss+    0:00 /sbin/agetty --noclear tty1 linux
 2582 ?        S      0:00 xterm -class UXTerm -title uxterm -u8
 2613 ?        Ss     0:00 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session x-terminal-emulator
 2616 ?        S      0:00 /usr/bin/dbus-launch --exit-with-session x-terminal-emulator
 2617 ?        Ss     0:00 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
 2686 pts/0    Ss+    0:00 bash
 3520 ?        Ss     0:00 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
 5504 ?        S      0:00 [NFSv4 callback]
 5517 ?        Ss     0:05 SCREEN
 5518 pts/2    Ss     0:00 /bin/bash
 5520 pts/2    S+     2:19 watch /srv/autobuilder/tmp/test-flock.py /srv/autobuilder/tmp/debian8
11207 ?        Sl     0:01 python3 ./bitbake/bin/bitbake-prserv --start
11217 ?        Sl     1:21 /usr/bin/python /home/pokybuild/yocto-autobuilder/bin/twistd --no_save -y buildbot.tac
11218 ?        S      0:00 /bin/sh -c python bin/buildworker-janitor /home/pokybuild/yocto-autobuilder/config/autobuilder.conf
11219 ?        Sl     0:00 python bin/buildworker-janitor /home/pokybuild/yocto-autobuilder/config/autobuilder.conf
12536 ?        Ss     0:00 sshd: rpurdie [priv]
12543 ?        S      0:00 sshd: rpurdie@pts/3 
12544 pts/3    Ss     0:00 -bash
13012 pts/3    R+     0:00 ps ax
25216 ?        Ss     0:15 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -c /var/lib/ntp/ntp.conf.dhcp -u 109:117
30827 ?        SN     0:16 nativepython /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/smart.real --log-level=warning --data-dir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/work/qemux86_64-poky-linux/core-image-lsb-dev/1.0-r0/rootfs/var/lib/smart install -y locale-base-en-us@core2_64 locale-base-en-gb@core2_64 packagegroup-base-extended@qemux86_64 smartpm@core2_64 psplash@core2_64 packagegroup-core-full-cmdline@all packagegroup-core-boot@qemux86_64 packagegroup-core-ssh-openssh@all rpm@core2_64 packagegroup-core-lsb@all
30833 ?        SN     0:15 nativepython /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/smart.real --log-level=warning --data-dir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/work/qemux86_64-poky-linux/core-image-lsb/1.0-r0/rootfs/var/lib/smart install -y locale-base-en-us@core2_64 locale-base-en-gb@core2_64 packagegroup-base-extended@qemux86_64 smartpm@core2_64 psplash@core2_64 packagegroup-core-full-cmdline@all packagegroup-core-boot@qemux86_64 packagegroup-core-ssh-openssh@all rpm@core2_64 packagegroup-core-lsb@all
30839 ?        SN     0:25 nativepython /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/smart.real --log-level=warning --data-dir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/work/qemux86_64-poky-linux/core-image-lsb-sdk/1.0-r0/rootfs/var/lib/smart install -y locale-base-en-us@core2_64 locale-base-en-gb@core2_64 packagegroup-base-extended@qemux86_64 psplash@core2_64 packagegroup-core-full-cmdline@all packagegroup-core-boot@qemux86_64 kernel-devsrc@qemux86_64 packagegroup-core-eclipse-debug@all packagegroup-core-tools-profile@qemux86_64 packagegroup-core-tools-testapps@qemux86_64 smartpm@core2_64 packagegroup-core-standalone-sdk-target@all packagegroup-core-tools-debug@all packagegroup-core-ssh-openssh@all packagegroup-core-sdk@qemux86_64 rpm@core2_64 packagegroup-core-lsb@all
35694 ?        S      0:00 /bin/sh -c . ./oe-init-build-env; checkvnc; if [ -d ../meta-selftest ]; then export DISPLAY=:1; oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg; else echo 'Skipping step - no meta-selftest layer here'; fi
35727 ?        Sl     0:01 python3 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg
36015 ?        Ss     0:05 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
36018 ?        Ss     0:05 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
40699 ?        SN     0:20 nativepython /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/smart.real --log-level=warning --data-dir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/work/genericx86_64-poky-linux/core-image-lsb/1.0-r0/rootfs/var/lib/smart install -y locale-base-en-us@core2_64 locale-base-en-gb@core2_64 packagegroup-base-extended@genericx86_64 rpm@core2_64 packagegroup-core-lsb@all packagegroup-core-boot@genericx86_64 packagegroup-core-ssh-openssh@all packagegroup-core-full-cmdline@all smartpm@core2_64 psplash@core2_64
41627 ?        SN     0:21 nativepython /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/smart.real --log-level=warning --data-dir=/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/work/genericx86_64-poky-linux/core-image-lsb-sdk/1.0-r0/rootfs/var/lib/smart install -y locale-base-en-us@core2_64 locale-base-en-gb@core2_64 packagegroup-core-tools-testapps@genericx86_64 smartpm@core2_64 packagegroup-core-tools-profile@genericx86_64 packagegroup-core-lsb@all packagegroup-core-boot@genericx86_64 psplash@core2_64 kernel-devsrc@genericx86_64 packagegroup-core-tools-debug@all packagegroup-base-extended@genericx86_64 packagegroup-core-eclipse-debug@all rpm@core2_64 packagegroup-core-sdk@genericx86_64 packagegroup-core-ssh-openssh@all packagegroup-core-full-cmdline@all packagegroup-core-standalone-sdk-target@all
46032 ?        S      0:00 /bin/sh -c EDITOR="echo 123" devtool -q edit-recipe pv
46034 ?        S      0:00 python3 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool -q edit-recipe pv
46066 ?        S      0:00 /bin/sh -c /usr/bin/vim "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/workspace/recipes/pv/pv_1.5.3.bb"
46067 ?        S      0:00 /usr/bin/vim /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/workspace/recipes/pv/pv_1.5.3.bb
53029 ?        Ss     0:06 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
53153 ?        Ss     0:05 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d
53214 ?        Ss     0:06 /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64-lsb/build/build/tmp/sysroots/x86_64-linux/usr/bin/pseudo -d

rpurdie@debian8:~$ pstree -p
systemd(1)─┬─PRServ(11207)───{PRServ Handler}(11208)
           ├─Xtightvnc(2560)
           ├─acpid(1753)
           ├─agetty(2578)
           ├─atd(1691)
           ├─cron(1689)
           ├─dbus-daemon(1699)
           ├─dbus-daemon(2617)
           ├─dbus-launch(2616)
           ├─dhclient(3520)
           ├─irqbalance(1763)
           ├─nativepython(30827)
           ├─nativepython(30833)
           ├─nativepython(30839)
           ├─nativepython(40699)
           ├─nativepython(41627)
           ├─ntpd(25216)
           ├─pseudo(36015)
           ├─pseudo(36018)
           ├─pseudo(53029)
           ├─pseudo(53153)
           ├─pseudo(53214)
           ├─rpc.idmapd(1686)
           ├─rpc.statd(1672)
           ├─rpcbind(1655)
           ├─rsyslogd(1749)─┬─{in:imklog}(1765)
           │                ├─{in:imuxsock}(1764)
           │                └─{rs:main Q:Reg}(1766)
           ├─screen(5517)───bash(5518)───watch(5520)
           ├─sh(11218)───python(11219)─┬─{python}(11221)
           │                           └─{python}(11222)
           ├─sshd(1690)───sshd(12536)───sshd(12543)───bash(12544)───pstree(13144)
           ├─systemd-journal(1253)
           ├─systemd-logind(1693)
           ├─systemd-udevd(1242)
           ├─twistd(11217)─┬─sh(35694)───python3(35727)─┬─sh(46032)───python3(46034)───sh(46066)───vim(46067)
           │               │                            └─{python3}(46033)
           │               └─{twistd}(11220)
           └─xstartup(2576)───xterm(2582)─┬─bash(2686)
                                          └─ssh-agent(2613)
Comment 1 Richard Purdie 2017-09-12 15:47:39 UTC
ssh-agent my have been from something else. Killing vim resulted in:

2017-09-11 23:54:38,035 - oe-selftest - INFO -   test_devtool_add (devtool.DevtoolTests)
2017-09-12 15:46:39,037 - oe-selftest - INFO -  ... FAIL (57121.003s)

and unblocked selftest:

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/511