Bug 1472

Summary: Failed to load root file system using NFS if board is connected to giga-bit network
Product: [Build System, Metadata & Runtime] BSPs Reporter: Adeel Arshad <adeel_arshad>
Component: bsps-configurationAssignee: Bruce Ashfield <bruce.ashfield>
Status: RESOLVED WONTFIX QA Contact:
Severity: minor    
Priority: Medium CC: adeel_arshad, drew, sgw, yp.bsp.watcher, yp.watcher
Version: 1.0   
Target Milestone: 1.2   
Hardware: Beagleboard   
OS: arm   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---
Attachments:
Description Flags
NFS traffic captured using wireshark on 10/100 mega network.
none
NFS traffic captured using wireshark on 1 giga network. none

Description Adeel Arshad 2011-09-13 02:58:55 UTC
Target Board : Beagle Board xM Rev B
Yocto repository : git://git.yoctoproject.org/poky

I am facing this issue when the board is connected to giga-bit network but this issue gets resolve when after some analysis I connected it to 10/100 mega-bit network. Following is the boot-log.


Texas Instruments X-Loader 1.5.0 (Aug 30 2011 - 17:23:36)
Beagle xM
Reading boot sector
Loading u-boot.bin from mmc


U-Boot 2011.06 (Aug 30 2011 - 17:23:41)

OMAP3630/3730-GP ES2.0, CPU-OPP2, L3-165MHz, Max CPU Clock 1 Ghz
OMAP3 Beagle board + LPDDR/NAND
I2C:   ready
DRAM:  512 MiB
NAND:  0 MiB
MMC:   OMAP SD/MMC: 0
*** Warning - readenv() failed, using default environment

In:    serial
Out:   serial
Err:   serial
Beagle xM Rev A
No EEPROM on expansion board
Die ID #727e00011ff00000015739eb0801d024
Hit any key to stop autoboot:  0 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # 
OMAP3 beagleboard.org # setenv bootargs 'console=ttyO2,115200n8 root=/dev/nfs rw nfsroot=137.202.157.192:/home/aarshad/tftpboot/beaglexm/nfs ip=dhcp nolock,rsize=1024,wsize=1024 rootdelay=2'
OMAP3 beagleboard.org # setenv bootcmd 'mmc rescan;fatload mmc 0 80300000 uImage;bootm 80300000'
OMAP3 beagleboard.org # boot
reading uImage

2809284 bytes read
## Booting kernel from Legacy Image at 80300000 ...
   Image Name:   Linux-2.6.37.6-yocto-standard+
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2809220 Bytes = 2.7 MiB
   Load Address: 80008000
   Entry Point:  80008000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.37.6-yocto-standard+ (aarshad@pkl-madeeha-ubuntu) (gcc version 4.6.1 20110627 (prerelease) (GCC) ) #1 PREEMPT Tue Sep 6 21:07:57 PKST 2011
CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: OMAP3 Beagle Board
Reserving 14680064 bytes SDRAM for VRAM
Memory policy: ECC disabled, Data cache writeback
OMAP3630 ES1.1 (l2cache iva sgx neon isp 192mhz_clk )
SRAM: Mapped pa 0x40200000 to va 0xfe400000 size: 0x10000
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 126464
Kernel command line: console=ttyO2,115200n8 root=/dev/nfs rw nfsroot=137.202.157.192:/home/aarshad/tftpboot/beaglexm/nfs ip=dhcp nolock,rsize=1024,wsize=1024 rootdelay=2
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
allocated 2621440 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Memory: 498MB = 498MB total
Memory: 496660k/496660k available, 27628k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
    vmalloc : 0xe0800000 - 0xf8000000   ( 376 MB)
    lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
    modules : 0xbf000000 - 0xc0000000   (  16 MB)
      .init : 0xc0008000 - 0xc0056000   ( 312 kB)
      .text : 0xc0056000 - 0xc0570b44   (5227 kB)
      .data : 0xc0572000 - 0xc05c7680   ( 342 kB)
SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Preemptable hierarchical RCU implementation.
NR_IRQS:402
Clocking rate (Crystal/Core/MPU): 26.0/332/600 MHz
Reprogramming SDRC clock to 332000000 Hz
GPMC revision 5.0
IRQ: Found an INTC at 0xfa200000 (revision 4.0) with 96 interrupts
Total of 96 interrupts on 1 active controller
OMAP clockevent source: GPTIMER12 at 32768 Hz
Console: colour dummy device 80x30
Calibrating delay loop... 496.77 BogoMIPS (lpj=1937408)
pid_max: default: 32768 minimum: 301
Security Framework initialized
Mount-cache hash table entries: 512
Initializing cgroup subsys debug
Initializing cgroup subsys ns
ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys blkio
CPU: Testing write buffer coherency: ok
ftrace: allocating 16434 entries in 33 pages
devtmpfs: initialized
regulator: core version 0.5
regulator: dummy: 
NET: Registered protocol family 16
OMAP GPIO hardware version 2.5
OMAP GPIO hardware version 2.5
OMAP GPIO hardware version 2.5
OMAP GPIO hardware version 2.5
OMAP GPIO hardware version 2.5
OMAP GPIO hardware version 2.5
omap_mux_init: Add partition: #1: core, flags: 0
OMAP3 Beagle Rev: xM A
Found NAND on CS0
Registering NAND on CS0
Unable to get DVI reset GPIO
beagle_opp_init: turbo OPPs enabled!
OMAP DMA hardware revision 5.0
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
omap_device: omap_i2c.1: new worst case activate latency 0: 30517
omap_i2c omap_i2c.1: bus 1 rev4.0 at 2600 kHz
twl4030: PIH (irq 7) chaining IRQs 368..375
twl4030: power (irq 373) chaining IRQs 376..383
twl4030: gpio (irq 368) chaining IRQs 384..401
regulator: VUSB1V5: 1500 mV normal standby
regulator: VUSB1V8: 1800 mV normal standby
regulator: VUSB3V1: 3100 mV normal standby
twl4030_usb twl4030_usb: Initialized TWL4030 USB module
regulator: VMMC1: 1850 <--> 3150 mV at 3000 mV normal standby
regulator: VDAC: 1800 mV normal standby
regulator: VDVI: 1800 mV normal standby
regulator: VSIM: 1800 <--> 3000 mV at 1800 mV normal standby
omap_i2c omap_i2c.2: bus 2 rev4.0 at 400 kHz
omap_i2c omap_i2c.3: bus 3 rev4.0 at 100 kHz
Advanced Linux Sound Architecture Driver Version 1.0.23.
Switching to clocksource 32k_counter
musb_hdrc: version 6.0, musb-dma, otg (peripheral+host), debug=0
omap_device: omap_i2c.1: new worst case deactivate latency 0: 30517
musb_hdrc musb_hdrc: USB OTG mode controller at fa0ab000 using DMA, IRQ 92
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 970
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
OMAP DSS rev 2.0
OMAP DISPC rev 3.0
OMAP VENC rev 2
OMAP DSI rev 1.0
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
omap_uart.0: ttyO0 at MMIO 0x4806a000 (irq = 72) is a OMAP UART0
omap_uart.1: ttyO1 at MMIO 0x4806c000 (irq = 73) is a OMAP UART1
omap_uart.2: ttyO2 at MMIO 0x49020000 (irq = 74) is a OMAP UART2
console [ttyO2] enabled
omap_uart.3: ttyO3 at MMIO 0x49042000 (irq = 80) is a OMAP UART3
brd: module loaded
omap2-nand driver initializing
No NAND device found.
No NAND device found.
usbcore: registered new interface driver kaweth
pegasus: v0.6.14 (2006/09/27), Pegasus/Pegasus II USB Ethernet driver
usbcore: registered new interface driver pegasus
usbcore: registered new interface driver asix
usbcore: registered new interface driver cdc_ether
usbcore: registered new interface driver dm9601
usbcore: registered new interface driver smsc75xx
usbcore: registered new interface driver smsc95xx
usbcore: registered new interface driver net1080
usbcore: registered new interface driver cdc_subset
usbcore: registered new interface driver zaurus
usbcore: registered new interface driver MOSCHIP usb-ethernet driver
usbcore: registered new interface driver int51x1
cdc_ncm: 30-Nov-2010
usbcore: registered new interface driver cdc_ncm
console [netcon0] enabled
netconsole: network logging started
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-omap.0 supply hsusb0 not found, using dummy regulator
ehci-omap.0 supply hsusb1 not found, using dummy regulator
ehci-omap ehci-omap.0: OMAP-EHCI Host Controller
ehci-omap ehci-omap.0: new USB bus registered, assigned bus number 1
ehci-omap ehci-omap.0: irq 77, io mem 0x48064800
ehci-omap ehci-omap.0: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
mice: PS/2 mouse device common for all mice
twl_rtc twl_rtc: rtc core: registered twl_rtc as rtc0
omap_device: omap_wdt.-1: new worst case activate latency 0: 30517
OMAP Watchdog Timer Rev 0x31: initial timeout 60 sec
cpuidle: using governor ladder
cpuidle: using governor menu
mmci-omap-hs: probe of mmci-omap-hs.1 failed with error -16
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
OMAP3 Beagle/Devkit8000 SoC init
asoc: twl4030-hifi <-> omap-mcbsp-dai.1 mapping ok
ALSA device list:
  #0: omap3beagle
TCP cubic registered
NET: Registered protocol family 17
Registering the dns_resolver key type
VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
ThumbEE CPU extension supported.
Power Management for TI OMAP3.
------------[ cut here ]------------
WARNING: at /home/aarshad/yocto-m3/yocto-poc/build/tmp/work/beagleboard-poky-linux-gnueabi/linux-yocto-2.6.37+git1+84f1a422d7e21fbc23a687035bdf9d42471f19e0_1+3ddb22772862a8223640fa97580569924f51bddc-r20/linu
x/arch/arm/mach-omap2/cpuidle34xx.c:466 omap_init_power_states+0x1b8/0x1d8()
omap_init_power_states: core off state C7 disabled due to i583
Modules linked in:
[<c0061540>] (unwind_backtrace+0x0/0xe8) from [<c008c9b0>] (warn_slowpath_common+0x54/0x6c)
[<c008c9b0>] (warn_slowpath_common+0x54/0x6c) from [<c008ca60>] (warn_slowpath_fmt+0x34/0x44)
[<c008ca60>] (warn_slowpath_fmt+0x34/0x44) from [<c006dce8>] (omap_init_power_states+0x1b8/0x1d8)
[<c006dce8>] (omap_init_power_states+0x1b8/0x1d8) from [<c0012d50>] (omap3_idle_init+0x50/0x168)
[<c0012d50>] (omap3_idle_init+0x50/0x168) from [<c0011c30>] (omap3_pm_init+0x538/0x764)
[<c0011c30>] (omap3_pm_init+0x538/0x764) from [<c00563e8>] (do_one_initcall+0xc0/0x194)
[<c00563e8>] (do_one_initcall+0xc0/0x194) from [<c00089c8>] (kernel_init+0xd0/0x1ac)
[<c00089c8>] (kernel_init+0xd0/0x1ac) from [<c005c1d4>] (kernel_thread_exit+0x0/0x8)
---[ end trace d1d264db2d72b8ed ]---
usb 1-2: new high speed USB device using ehci-omap and address 2
sr_init: No PMIC hook to init smartreflex
smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
SmartReflex Class3 initialized
clock: disabling unused clocks to save power
registered taskstats version 1
Console: switching to colour frame buffer device 80x30
regulator_init_complete: incomplete constraints, leaving VDAC on
twl_rtc twl_rtc: setting system clock to 2011-09-01 18:43:59 UTC (1314902639)
mmc0: host does not support reading read-only switch. assuming write-enable.
mmc0: new SDHC card at address e624
mmcblk0: mmc0:e624 SU04G 3.69 GiB 
 mmcblk0: p1 p2
hub 1-2:1.0: USB hub found
hub 1-2:1.0: 5 ports detected
usb 1-2.1: new high speed USB device using ehci-omap and address 3
smsc95xx v1.0.4
smsc95xx 1-2.1:1.0: usb0: register 'smsc95xx' at usb-ehci-omap.0-2.1, smsc95xx USB 2.0 Ethernet, f6:3c:87:a9:88:e8
Sending DHCP requests .
usb0: link up, 100Mbps, full-duplex, lpa 0xC5E1
., OK
IP-Config: Got DHCP answer from 0.0.0.0, my address is 137.202.157.152
IP-Config: Complete:
     device=usb0, addr=137.202.157.152, mask=255.255.254.0, gw=137.202.157.254,
     host=137.202.157.152, domain=mgc.mentorg.com, nis-domain=(none),
     bootserver=0.0.0.0, rootserver=137.202.157.192, rootpath=
Waiting 2sec before mounting root device...
VFS: Mounted root (nfs filesystem) on device 0:15.
devtmpfs: mounted
Freeing init memory: 312K
nfs: server 137.202.157.192 not responding, still trying
nfs: server 137.202.157.192 not responding, still trying
Comment 1 Bruce Ashfield 2011-09-13 11:41:55 UTC
strange. And if this is plugged into a 100Meg network it boots ?
I see it negotiating down to 100 in the logs.

  usb0: link up, 100Mbps, full-duplex, lpa 0xC5E1., OK

Can you provide some information on the route to the server ? This really looks like infrastructure from here.
Comment 2 Adeel Arshad 2011-09-14 00:48:45 UTC
(In reply to comment #1)
> strange. And if this is plugged into a 100Meg network it boots ?
> I see it negotiating down to 100 in the logs.
> 
>   usb0: link up, 100Mbps, full-duplex, lpa 0xC5E1., OK
> 
> Can you provide some information on the route to the server ? This really looks
> like infrastructure from here.

Yea It boots/loads rootfs successfully when plugged into 10/100Meg Network.
And regarding the Route, I just replaced the TP-LINK Gigabit Ethernet Switch with D-LINK 10/100 Fast Ethernet Switch. No change is route to server. Also the "target board" and the "host server" are on the same network.
Comment 3 Bruce Ashfield 2011-09-14 09:26:10 UTC
Hmm. So it sounds like we need to find out who's dropping these
packets. 

I don't have a gige network to plug into here for my own tests, so if I can
impose an extra request. Do you know if a flash based boot and another
network transfer suffers these issues ? Or are we having some issue with
certain types of traffic, etc.
Comment 4 Adeel Arshad 2011-09-20 04:04:33 UTC
Created attachment 240 [details]
NFS traffic captured using wireshark on 10/100 mega network.
Comment 5 Adeel Arshad 2011-09-20 04:15:37 UTC
Created attachment 241 [details]
NFS traffic captured using wireshark on 1 giga network.
Comment 6 Adeel Arshad 2011-09-20 04:30:34 UTC
(In reply to comment #3)
> Hmm. So it sounds like we need to find out who's dropping these
> packets. 
> 
> I don't have a gige network to plug into here for my own tests, so if I can
> impose an extra request. Do you know if a flash based boot and another
> network transfer suffers these issues ? Or are we having some issue with
> certain types of traffic, etc.

ok I think this will consume more time, but anyhow I have more information for you. I have attached the wireshark/ethereal packet capture to provide you more insight that how the NFS traffic flow is disturbed in case of gigabit network. My observation is that the problem is related to the packet size/length instead of traffic type, as even in case of NFS traffic the initial communication is successful. The file containing the traffic flow on gigabit network is attached successfully but as file containing the traffic flow on megabit network is around 3.4 MB so I am not able to attach it here. Let me know if u need that file as well, which can help you in the comparison of traffic flow.( gigabit vs megabit ) .
Comment 7 Bruce Ashfield 2011-09-21 12:27:37 UTC
Interesting. Have you tried any other variations on rsize=1024,wsize=1024 ?
Comment 8 Adeel Arshad 2011-09-27 01:24:33 UTC
(In reply to comment #7)
> Interesting. Have you tried any other variations on rsize=1024,wsize=1024 ?

Yea I have tested with rsize=1024,wsize=1024 in the bootargs, and yes it is able to load rootfs using NFS successfully on Gigabit network. So limiting the packet size is working.
Comment 9 Adeel Arshad 2011-10-03 00:14:45 UTC
I hope I have provided enough information to change the status from needinfo -> open.
Comment 10 Bruce Ashfield 2011-10-03 06:16:34 UTC
We can leave it open, but there's almost nothing we can do with this. At least
not kernel size. The fact that this doesn't work on one network versus the other
and then works when the read/write size is restricted indicates that something
in the GigE network is not passing data over a particular size.

This is what we call and infrastructure issue, and there's nothing we can really
do with it from the BSP/kernel. Something in startup/userspace could be 
tweaked to make this work out of the box, but that's about it at the moment.
Comment 11 Bruce Ashfield 2011-12-15 11:40:54 UTC
I'm tagging this as won't fix, since there is an acceptable boot parameter fix for the issue, and the issue itself is depending on the network being used.