Bug 8550

I was able to reproduce it again by pure luck and found the same thing, the route wasn't in the host routing table.

I checked the runqemu-ifup and there are no checks for the IFCONFIG or IPTABLES commands. A first approach is to have checks here.

Another approach is in the ping test to have check the host routing table before doing the ping.

Sent a couple of patches yesterday to have the output from runqemu scripts, and to verify if a interface was created correctly. This won't solve the bug but will give us a better view.

Checking a bit more errors.yoctoproject.org, the vast majority of the ping failures aren't from yoctoproject.org ABs infrastructure; most of them are from a Fedora 22 host, but we can't verify the set up.

Created attachment 2820 [details]
log testimage 2015-10-23

Created attachment 2821 [details]
low qemuboot 2015-10-23

These logs files are from a failure from yesterday.

The autobuilder link is:

https://autobuilder.yoctoproject.org/main/builders/nightly-multilib/builds/523/steps/Running%20Sanity%20Tests_2/logs/stdio

Created attachment 2822 [details]
dumps 2015-10-23

Dumps from the same test

It seems the problem is with connman; it assign another IP address. Here are snips from different logs:

-------------------- snip  ip addr from target -----------------

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP8000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet 169.254.159.3/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe12:3456/64 scope link 
       valid_lft forever preferred_lft forever



-------------------- snip  /var/log/messages from target -----------------

Oct 22 02:38:30 qemux86-64 daemon.info avahi-daemon[746]: Joining mDNS multicast group on interface eth0.IPv4 with address 169.254.159.3.
Oct 22 02:38:30 qemux86-64 daemon.info avahi-daemon[746]: New relevant interface eth0.IPv4 for mDNS.
Oct 22 02:38:30 qemux86-64 daemon.info avahi-daemon[746]: Registering new address record for 169.254.159.3 on eth0.IPv4.
Oct 22 02:38:30 qemux86-64 daemon.info connmand[672]: eth0 {add} address 169.254.159.3/16 label eth0 family 2
Oct 22 02:38:30 qemux86-64 daemon.info connmand[672]: eth0 {add} route 169.254.0.0 gw 0.0.0.0 scope 253 <LINK>
Oct 22 02:38:30 qemux86-64 daemon.info connmand[672]: eth0 {add} route 0.0.0.0 gw 0.0.0.0 scope 253 <LINK>



-------------------- qemuboot snip ----------------------------------------

qemux86-64 login: 
(connman-applet:779): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache': No such file or directory

This likely means that your installation is broken.
Try running the command
  gdk-pixbuf-query-loaders > /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
to make things work again for the time being.
matchbox: Cant find a keycode for keysym 269025056
matchbox: ignoring key shortcut XF86Calendar=!$contacts

matchbox: Cant find a keycode for keysym 2809
matchbox: ignoring key shortcut telephone=!$dates

matchbox: Cant find a keycode for keysym 269025050
matchbox: ignoring key shortcut XF86Start=!matchbox-remote -desktop

[settings daemon] Forking. run with -n to prevent fork

** (matchbox-panel:810): WARNING **: Failed to load applet "battery" (/usr/lib64/matchbox-panel/libbattery.so: cannot open shared object file: No such file or directory).

** (connman-applet:779): WARNING **: Missing icon connman-connecting-01: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-02: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-03: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-04: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-05: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-06: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-07: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-08: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-09: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-10: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-11: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-12: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-13: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-14: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-15: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-16: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-17: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-18: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-19: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-20: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-21: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-22: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-23: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-24: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-25: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-26: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-27: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-28: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-29: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-30: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-31: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-32: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-connecting-33: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-signal-01: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-signal-02: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-signal-03: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-signal-04: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-signal-05: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-type-none: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-type-wired: Unrecognized image file format

** (connman-applet:779): WARNING **: Missing icon connman-notifier-unavailable: Unrecognized image file format

(connman-applet:779): Gtk-WARNING **: Unable to locate theme engine in module_path: "sato-engine",

After checking the connman's IPKs, the icons are there and the configuration is there. It seems the problem is not when generating the package but in runtime.

(In reply to comment #9)
> After checking the connman's IPKs, the icons are there and the configuration
> is there. It seems the problem is not when generating the package but in
> runtime.


As Richard  pointed out in email:

> The problem is that it installs lib32-gdk-pixbuf and lib32-libgtk+
> however it doesn't install any lib32 gdk-pixbuf loaders.
...
> FWIW, I did try:
>
> IMAGE_INSTALL_append = " lib32-connman-gnome lib32-gdk-pixbuf-loader-png lib32-gdk-pixbuf-loader-jpeg"
>
> however whilst this fixed the connman-applet start errors, the icons on
> the desktop disappeared, I worry that the icon cache format is multilib
> specific and you can't run two different sized libgtk on the same icon
> directory.

so those applet warnings are pixbuf problems (or package management, depending how you look at it).

However, I don't understand how that makes connman itself not work? Is applet actually doing something to enable DHCP -- I would have guessed no but I've never looked at the applet before this...

Filed bug 8570 for the gdk-pixbuf issues, on the assumption that the issues are not the same: let's make it a duplicate if that turns out false.

There are another two testimage fails with the multilib image. In these tests the ping doesn't fail, but fails other tests. Checking the logs for these failures it shows there aren't gdk-pixbuf/connman errors. So it seems the ping failure is related with the gdk-pixbuf problems.

A way to reproduce the problem is tu build core-image-sato with the next parameters:

require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
RPM_PREFER_ELF_ARCH = "1"
IMAGE_INSTALL_append = " lib32-connman-gnome"
MACHINE = "qemux86-64"

With the previous parameters the problem is 100% reproducible and persist between boots. I debug the running image and found the problem is with the "wired-setup" file; this file is generated with the connman-conf recipe and is used to have connectivity with the qemu images.

The problem is that the installed init script search for this file in /usr/lib/connman/wired-setup, but the file is installed in /usr/lib64/connman/wired-setup. Checking the paths in both recipes we found both are using libdir, but it seems the libdir is not the same for both recipes.

connman init script:
{LIBDIR}/connman/wired-setup

conmman-conf recipe:
install -m 0755 ${WORKDIR}/wired-setup ${D}${libdir}/connman

Rebuild with core-image-sato sstates and the initscript has the correct path to "/usr/lib64/connman/wired-setup". Used cleanall with the connman recipe and tried again and the init script was using the correct path.

(In reply to comment #14)
> The problem is that the installed init script search for this file in
> /usr/lib/connman/wired-setup, but the file is installed in
> /usr/lib64/connman/wired-setup. Checking the paths in both recipes we found
> both are using libdir, but it seems the libdir is not the same for both
> recipes.

Good find!

wired-setup is not lib-specific so IMO should really be in {datadir}/connman/wired-setup. I can send a patch for that.

Filed Bug 8586 - Multilib build includes 32bit connman without good reason.

commit 63a0311497a0f9d3a748da254b631d3c7edc1364
Author: Jussi Kukkonen <jussi.kukkonen@intel.com>
Date:   Tue Oct 27 15:16:40 2015 +0200

    connman: Move wired-setup to ${datadir}
    
    wired-setup script should not be in ${libdir} as it's not arch
    dependent.
    
    This also fixes (or works around) a practical issue where a multilib
    build installs the wrong version of connman-conf and then connman
    can't find the script.
    
    [YOCTO #8550]
    
    (From OE-Core rev: 38a6ecf0070a60eb14b353b158b70ddc919ad328)
    
    Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Patch is now in master:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=63a0311497a0f9d3a748da254b631d3c7edc1364

Verified on commit 370a2916c4559c17952392b872322e8e490b124c