There are several ping failures in http://errors.yoctoproject.org. Some ssh tests aren't skipped if the ping fails and there are some interesting traceback in the output. Most of the time it is "time out", but other recurring error is "no route to host". There is a lot of ping errors, but there isn't a lot of information to know why it failed. I was troubleshooting one of these today and found the logs and dumps didn't help me at all to debug this problem. I had access to the build machine and tried to use the runqemu script to see what happened; in this particular case, the route wasn't added to the host routing table. Checking a little bit more, a way to reproduce the issue is to create a tap interface but don't add any route to the routing table, then make runqemu use this tap interface. Of course this doesn't happen during normal operation of runqemu script.
I was able to reproduce it again by pure luck and found the same thing, the route wasn't in the host routing table. I checked the runqemu-ifup and there are no checks for the IFCONFIG or IPTABLES commands. A first approach is to have checks here.
Another approach is in the ping test to have check the host routing table before doing the ping.
Sent a couple of patches yesterday to have the output from runqemu scripts, and to verify if a interface was created correctly. This won't solve the bug but will give us a better view.
Checking a bit more errors.yoctoproject.org, the vast majority of the ping failures aren't from yoctoproject.org ABs infrastructure; most of them are from a Fedora 22 host, but we can't verify the set up.
Created attachment 2820 [details] log testimage 2015-10-23
Created attachment 2821 [details] low qemuboot 2015-10-23 These logs files are from a failure from yesterday. The autobuilder link is: https://autobuilder.yoctoproject.org/main/builders/nightly-multilib/builds/523/steps/Running%20Sanity%20Tests_2/logs/stdio
Created attachment 2822 [details] dumps 2015-10-23 Dumps from the same test
It seems the problem is with connman; it assign another IP address. Here are snips from different logs: -------------------- snip ip addr from target ----------------- 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP8000> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 169.254.159.3/16 brd 169.254.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever -------------------- snip /var/log/messages from target ----------------- Oct 22 02:38:30 qemux86-64 daemon.info avahi-daemon[746]: Joining mDNS multicast group on interface eth0.IPv4 with address 169.254.159.3. Oct 22 02:38:30 qemux86-64 daemon.info avahi-daemon[746]: New relevant interface eth0.IPv4 for mDNS. Oct 22 02:38:30 qemux86-64 daemon.info avahi-daemon[746]: Registering new address record for 169.254.159.3 on eth0.IPv4. Oct 22 02:38:30 qemux86-64 daemon.info connmand[672]: eth0 {add} address 169.254.159.3/16 label eth0 family 2 Oct 22 02:38:30 qemux86-64 daemon.info connmand[672]: eth0 {add} route 169.254.0.0 gw 0.0.0.0 scope 253 <LINK> Oct 22 02:38:30 qemux86-64 daemon.info connmand[672]: eth0 {add} route 0.0.0.0 gw 0.0.0.0 scope 253 <LINK> -------------------- qemuboot snip ---------------------------------------- qemux86-64 login: (connman-applet:779): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache': No such file or directory This likely means that your installation is broken. Try running the command gdk-pixbuf-query-loaders > /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache to make things work again for the time being. matchbox: Cant find a keycode for keysym 269025056 matchbox: ignoring key shortcut XF86Calendar=!$contacts matchbox: Cant find a keycode for keysym 2809 matchbox: ignoring key shortcut telephone=!$dates matchbox: Cant find a keycode for keysym 269025050 matchbox: ignoring key shortcut XF86Start=!matchbox-remote -desktop [settings daemon] Forking. run with -n to prevent fork ** (matchbox-panel:810): WARNING **: Failed to load applet "battery" (/usr/lib64/matchbox-panel/libbattery.so: cannot open shared object file: No such file or directory). ** (connman-applet:779): WARNING **: Missing icon connman-connecting-01: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-02: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-03: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-04: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-05: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-06: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-07: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-08: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-09: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-10: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-11: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-12: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-13: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-14: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-15: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-16: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-17: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-18: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-19: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-20: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-21: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-22: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-23: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-24: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-25: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-26: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-27: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-28: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-29: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-30: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-31: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-32: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-connecting-33: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-signal-01: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-signal-02: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-signal-03: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-signal-04: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-signal-05: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-type-none: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-type-wired: Unrecognized image file format ** (connman-applet:779): WARNING **: Missing icon connman-notifier-unavailable: Unrecognized image file format (connman-applet:779): Gtk-WARNING **: Unable to locate theme engine in module_path: "sato-engine",
After checking the connman's IPKs, the icons are there and the configuration is there. It seems the problem is not when generating the package but in runtime.
(In reply to comment #9) > After checking the connman's IPKs, the icons are there and the configuration > is there. It seems the problem is not when generating the package but in > runtime. As Richard pointed out in email: > The problem is that it installs lib32-gdk-pixbuf and lib32-libgtk+ > however it doesn't install any lib32 gdk-pixbuf loaders. ... > FWIW, I did try: > > IMAGE_INSTALL_append = " lib32-connman-gnome lib32-gdk-pixbuf-loader-png lib32-gdk-pixbuf-loader-jpeg" > > however whilst this fixed the connman-applet start errors, the icons on > the desktop disappeared, I worry that the icon cache format is multilib > specific and you can't run two different sized libgtk on the same icon > directory. so those applet warnings are pixbuf problems (or package management, depending how you look at it). However, I don't understand how that makes connman itself not work? Is applet actually doing something to enable DHCP -- I would have guessed no but I've never looked at the applet before this...
Filed bug 8570 for the gdk-pixbuf issues, on the assumption that the issues are not the same: let's make it a duplicate if that turns out false.
There are another two testimage fails with the multilib image. In these tests the ping doesn't fail, but fails other tests. Checking the logs for these failures it shows there aren't gdk-pixbuf/connman errors. So it seems the ping failure is related with the gdk-pixbuf problems.
A way to reproduce the problem is tu build core-image-sato with the next parameters: require conf/multilib.conf MULTILIBS = "multilib:lib32" DEFAULTTUNE_virtclass-multilib-lib32 = "x86" RPM_PREFER_ELF_ARCH = "1" IMAGE_INSTALL_append = " lib32-connman-gnome" MACHINE = "qemux86-64"
With the previous parameters the problem is 100% reproducible and persist between boots. I debug the running image and found the problem is with the "wired-setup" file; this file is generated with the connman-conf recipe and is used to have connectivity with the qemu images. The problem is that the installed init script search for this file in /usr/lib/connman/wired-setup, but the file is installed in /usr/lib64/connman/wired-setup. Checking the paths in both recipes we found both are using libdir, but it seems the libdir is not the same for both recipes. connman init script: {LIBDIR}/connman/wired-setup conmman-conf recipe: install -m 0755 ${WORKDIR}/wired-setup ${D}${libdir}/connman
Rebuild with core-image-sato sstates and the initscript has the correct path to "/usr/lib64/connman/wired-setup". Used cleanall with the connman recipe and tried again and the init script was using the correct path.
(In reply to comment #14) > The problem is that the installed init script search for this file in > /usr/lib/connman/wired-setup, but the file is installed in > /usr/lib64/connman/wired-setup. Checking the paths in both recipes we found > both are using libdir, but it seems the libdir is not the same for both > recipes. Good find! wired-setup is not lib-specific so IMO should really be in {datadir}/connman/wired-setup. I can send a patch for that.
Filed Bug 8586 - Multilib build includes 32bit connman without good reason.
commit 63a0311497a0f9d3a748da254b631d3c7edc1364 Author: Jussi Kukkonen <jussi.kukkonen@intel.com> Date: Tue Oct 27 15:16:40 2015 +0200 connman: Move wired-setup to ${datadir} wired-setup script should not be in ${libdir} as it's not arch dependent. This also fixes (or works around) a practical issue where a multilib build installs the wrong version of connman-conf and then connman can't find the script. [YOCTO #8550] (From OE-Core rev: 38a6ecf0070a60eb14b353b158b70ddc919ad328) Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patch is now in master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=63a0311497a0f9d3a748da254b631d3c7edc1364
Verified on commit 370a2916c4559c17952392b872322e8e490b124c