Bug 3938

Hello Yi,

Please state the exact issue, because I did not understand the exact issue.
Let me explain:
I have reviewed the commit 9eb88ceb39b7d0b8ddc6487e61ce8edadef10ec4 and it seems OK up to some point.
I don't have how to test that, because we don't have boards with two ethernet interfaces around.
But is seems that the patch will make connman ignore eth0, in order to keep previous settings for eth0.
Now, if at ifconfig, eth1 is the first interface returned, the patch can create trouble indeed.
Can you please run the ifconfig expression from the patch and post the results for routerstation and mpc8315e?
Now, onto the second issue at hand.
I am not familiar enough with connman on how it should deal with disconnected ethernet interfaces. But if eth1 is seen as up by connman (which seems to be the case for eth1, judging by your report), the address set on eth1 is a legit one, as connman will fail to set an address via DHCP and will assign the interface a Zeroconf IP address (starting with 169.254.xxx.xxx), which seems to be the case.
That's why I asked what is the exact issue.
Please help on this one.

Regards,
Cristian

(In reply to comment #1)
> Hello Yi,
> 
> Please state the exact issue, because I did not understand the exact issue.
> Let me explain:
> I have reviewed the commit 9eb88ceb39b7d0b8ddc6487e61ce8edadef10ec4 and it
> seems OK up to some point.
> I don't have how to test that, because we don't have boards with two
> ethernet interfaces around.
> But is seems that the patch will make connman ignore eth0, in order to keep
> previous settings for eth0.
> Now, if at ifconfig, eth1 is the first interface returned, the patch can
> create trouble indeed.
> Can you please run the ifconfig expression from the patch and post the
> results for routerstation and mpc8315e?

After the system boot up, run the command would get the following message:
#################
root@routerstationpro:~# ifconfig | grep eth | sed -e "s/\(eth[0-9]\)\(.*\)/\1/"
eth0
eth1
root@routerstationpro:~# 
#################

#################
root@mpc8315e-rdb:~#  ifconfig | grep eth | sed -e "s/\(eth[0-9]\)\(.*\)/\1/"
eth0
eth1
root@mpc8315e-rdb:~#
#################

> Now, onto the second issue at hand.
> I am not familiar enough with connman on how it should deal with
> disconnected ethernet interfaces. But if eth1 is seen as up by connman
> (which seems to be the case for eth1, judging by your report), the address
> set on eth1 is a legit one, as connman will fail to set an address via DHCP
> and will assign the interface a Zeroconf IP address (starting with
> 169.254.xxx.xxx), which seems to be the case.
> That's why I asked what is the exact issue.
> Please help on this one.

I found on routerstation, if eth1 is set with an ip address, I can not ping or ssh to the target via eth0. I must ifconfig down eth1, then it works.  

For mpc8315e, although the eth1 is also up, but there is no ip address with it. I can ping or ssh to it via eth0.  

> 
> Regards,
> Cristian

Here is the route message on routerstation:
###############
root@routerstationpro:~# route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         *               0.0.0.0         U     0      0        0 eth1
default         128.224.165.1   0.0.0.0         UG    0      0        0 eth0
128.224.165.0   *               255.255.255.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth1
root@routerstationpro:~# 
root@routerstationpro:~# 
root@routerstationpro:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 eth1
0.0.0.0         128.224.165.1   0.0.0.0         UG    0      0        0 eth0
128.224.165.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
root@routerstationpro:~#
###############


After eth1 is down, the route message :
###############
root@routerstationpro:~# ifconfig eth1 down
eth1: link down
root@routerstationpro:~# ADDRCONF(NETDEV_UP): eth1: link is not ready

root@routerstationpro:~# route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         128.224.165.1   0.0.0.0         UG    0      0        0 eth0
128.224.165.0   *               255.255.255.0   U     0      0        0 eth0
root@routerstationpro:~# 
root@routerstationpro:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         128.224.165.1   0.0.0.0         UG    0      0        0 eth0
128.224.165.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
root@routerstationpro:~# 
################

The routing tables don't look right. All interfaces have metric 0.
Yi, how are those routing tables generated? Are there any changes done to the BSPs?

(In reply to comment #4)
> The routing tables don't look right. All interfaces have metric 0.
> Yi, how are those routing tables generated? Are there any changes done to
> the BSPs?

The route of eth1 is set by conman, I can find the log in /var/log/messages:
###################
238 Mar  6 00:29:43 routerstationpro daemon.info connmand[178]: eth1 {add} address 169.254.184.164/16 label eth1 family 2
239 Mar  6 00:29:43 routerstationpro daemon.info connmand[178]: eth1 {add} route 169.254.0.0 gw 0.0.0.0 scope 253 <LINK>
240 Mar  6 00:29:43 routerstationpro daemon.info connmand[178]: eth1 {add} route 0.0.0.0 gw 0.0.0.0 scope 253 <LINK>
241 Mar  6 00:29:43 routerstationpro daemon.err connmand[178]: Setting default interface route failed (File exists)
#####################

The rote of eth0 is set by cmdline when booting from red-boot:
###########
root@routerstationpro:~# cat /proc/cmdline 
  console=ttyS0,115200 root=/dev/nfs rw nfsroot=128.224.178.20:/export/pxeboot/vlm-boards/19255/rootfs ip=128.224.178.127:128.224.178.20:128.224.178.1:255.255.254.0::eth0:off board=UBNT-RSPRO ethaddr=00.00.00.00.46.61
###########

It looks there are 2 default gws in route table:
#################
root@routerstationpro:~# route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         *               0.0.0.0         U     0      0        0 eth1
default         128.224.178.1   0.0.0.0         UG    0      0        0 eth0
128.224.178.0   *               255.255.254.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth1
##################

But the default dev is eth1:
###################
root@routerstationpro:~# ip route
default dev eth1 
default via 128.224.178.1 dev eth0 
128.224.178.0/23 dev eth0  src 128.224.178.127 
169.254.0.0/16 dev eth1  src 169.254.184.164 
root@routerstationpro:~# 
####################

So when I connect the remote target, it will try to route from eth1. The traceroute shows the following message:
##################
root@routerstationpro:~# traceroute 128.224.162.212
traceroute to 128.224.162.212 (128.224.162.212), 30 hops max, 38 byte packets
 1  routerstationpro.local (169.254.184.164)  2991.308 ms !H  2994.546 ms !H  2999.787 ms !H
root@routerstationpro:~#
#################


There aren't any changes for the system. This issue happens after the patch 9eb88ceb39b7d0b8ddc6487e61ce8edadef10ec4 applying.

Hello Yi,

As I said previously, it is quite hard for me to reproduce this issue.
Yi, if I will send you some patches to test, can you help me with testing?
That's one thing.

The second would be:
For this board, how eth0 network interface gets its IP address?

Static IP via bootloader (kernel command line parameters)?
Or DHCP? Or another way (please describe)?

I might have a proper fix now that connman supports ethernet provisioning.

Regards,
Cristian

(In reply to comment #6)
> Hello Yi,
> 
> As I said previously, it is quite hard for me to reproduce this issue.
> Yi, if I will send you some patches to test, can you help me with testing?
> That's one thing.

It's ok for me.

> 
> The second would be:
> For this board, how eth0 network interface gets its IP address?
> 
> Static IP via bootloader (kernel command line parameters)?
> Or DHCP? Or another way (please describe)?
> 

The static ip of eth0 is set in kernel command line by bootloader

> I might have a proper fix now that connman supports ethernet provisioning.
> 
> Regards,
> Cristian

Hello.

At this point, I don't have a complete fix to this issue.

My line of action would be to do something similar to what I implemented to qemu machines that use connman to take care of networking.

However, I recently found out that my solution does not work with systemd, so a redesign is needed.

Regards,
Cristian

Redesign for systemd init system is complete, waiting for review and integration.

Implementation started.

Retarget to 1.5, too invasive for 1.4

In order to choose the right technical solution, this needs to be discussed with the connman authors.

Hello Yi,

What's the status of this issue?
I am getting back working at it after a long hiatus.
What's the Yocto image that are qorking with?
Is core-image-sato?

Can you also please attach a complete /var/log/messages for the routerstation board?

Hello Yi,
Also, for routerstation, please let me know how is eth1 brought up (if anything special is done).

Hy Yi.
Can you also please re-test this issue and report the outcome?

Created attachment 1455 [details]
message log

(In reply to comment #13)
> Hello Yi,
> 

Hi Cristian,

> What's the status of this issue?

This issue still exists.


> I am getting back working at it after a long hiatus.
> What's the Yocto image that are qorking with?
> Is core-image-sato?
> 

It is core-image-sato-sdk. 

> Can you also please attach a complete /var/log/messages for the
> routerstation board?

I attached the full message log.

(In reply to comment #14)
> Hello Yi,
> Also, for routerstation, please let me know how is eth1 brought up (if
> anything special is done).

The eth1 is brought up by connmand.
Here is the reason:

The start function in /etc/init.d/connman:
################
do_start() {
    EXTRA_PARAM=""
    if test $nfsroot -eq 1 ; then
        ethn=`ifconfig | grep eth | sed -e "s/\(eth[0-9]\)\(.*\)/\1/"`
        EXTRA_PARAM="-I $ethn"
    fi  
    if [ -f /usr/lib/connman/wired-setup ] ; then
        . /usr/lib/connman/wired-setup
    fi  
    $DAEMON $EXTRA_PARAM
}
################

When the board boot up, the kernel detects eth0 and eth1 on the board. So the $ethn should be "eth0 eth1" (It is a nfs boot).  The "$DAEMON $EXTRA_PARAM" extends to "/usr/sbin/connmand -I eth0 eth1". But this is not a legal argument. The eth1 will be ignored with -I option. Then the eth1 still setup by connmand. If we want to ignore both eth0 and eth1, it should be "$DAEMON -I eth0,eth1"

This issue is introduced by commit 9eb88ceb39b7d0b8ddc6487e61ce8edadef10ec4. And the author also said: it might not work if using more than one eth interface. But it's better than not using connman with nfs at all.

Hello Yi,

I've talked with Cristi about the problem and we found the following:

1) For connman to ignore both the interfaces a comma should be used. However only the details for the eth0 interfaces are sent through the cmdline. Considering this, eth0 should be the only one ignored. A fix can be done to ignore only the interfaces that are specified in the cmdline.

2) From the log, it is quite clear that the eth1 interface link is up. 

Sep  4 15:00:03 routerstationpro user.info kernel: eth1: link up (1000Mbps/Full duplex)
Sep  4 15:00:03 routerstationpro daemon.info connmand[403]: eth1 {add} route ff00:: gw :: scope 0 <UNIVERSE>
Sep  4 15:00:03 routerstationpro daemon.info connmand[403]: eth1 {add} route fe80:: gw :: scope 0 <UNIVERSE>
Sep  4 15:00:03 routerstationpro daemon.info connmand[403]: eth1 {update} flags 102467 <UP,RUNNING,LOWER_UP>
Sep  4 15:00:03 routerstationpro daemon.info connmand[403]: eth1 {newlink} index 3 address 02:00:00:00:46:61 mtu 1500
Sep  4 15:00:03 routerstationpro daemon.info connmand[403]: eth1 {newlink} index 3 operstate 6 <UP>
Sep  4 15:00:03 routerstationpro user.info kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

connman intercepts the newlink signal and assigns the interface a zeroconf IP because it doesn't have DHCP or a static IP. So this is the expected behavior. A possible solution would be to include a custom script in your BSP that handles this behavior and ignore eth1.

3) From the log it seems like the eth1 interface is in an internal switch:

Sep  4 15:00:02 routerstationpro user.info kernel: eth1: Atheros AG71xx at 0xba000000, irq 5
Sep  4 15:00:02 routerstationpro user.info kernel: eth1: Atheros AR8316 switch driver attached.
Sep  4 15:00:02 routerstationpro user.info kernel: ar8316: Using port 4 as PHY
Sep  4 15:00:02 routerstationpro user.info kernel: ag71xx ag71xx.1 eth1: connected to PHY at ag71xx-mdio.0:00 [uid=004dd041, driver=Atheros AR8216/AR8236/AR8316]

This is a possible reason for which the interface is up.

Considering all of this, I can implement the fix specified at 1), but that won't solve the problems with the routes. The real problem is that the eth1 interface link is up eventough it's unused, but this is not because to connman.

Jukka Rissanen has sent a fix for the problem that I described in my other post with the incorrect ignoring of the interfaces.

See http://lists.openembedded.org/pipermail/openembedded-core/2013-September/083958.html for details.

Jukka Rissanen's patch has been merged to master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=c73487302270a7c2d3bbbb191ce1dd63ee2016dc