Bug 6732 - USB 2.0 Network interface causes firmware to stall on boot
Summary: USB 2.0 Network interface causes firmware to stall on boot
Status: RESOLVED FIXED
Alias: None
Product: MinnowBoard MAX Firmware
Classification: Hardware Platforms
Component: minnowboard-uefi-firmware (show other bugs)
Version: 2C A1
Hardware: MinnowBoard Max x86_64
: Medium+ critical
Target Milestone: Production Release
Assignee: John 'Warthog9' Hawley
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2014-09-18 00:53 UTC by John 'Warthog9' Hawley
Modified: 2015-03-31 23:14 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Enhanced error handling of XHCI control transfer (4.24 MB, application/x-zip-compressed)
2015-03-03 09:25 UTC, David_Wei
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description John 'Warthog9' Hawley 2014-09-18 00:53:00 UTC
I've got a Trendnet TU2-ET100 USB2.0 network interface based on the ASIX AX88772 chipset, plugged into the USB 2.0 port, and the MAX connected to a 5v 2.5a power supply, hdmi and serial console.

On boot I get the debugging output to the serial console that CircuitCo requested, however I get nothing else after that.

Unplugging the device and letting the firmware start, and while it's at the 5 second timeout screen plugging it in cause the firmware to stop and become unresponsive.

Plugging the device in after it's handed over to Linux everything works as expected.

I've confirmed this on two boards, one with firmware 0.71 and one with 0.73, both A1 boards.
Comment 1 John 'Warthog9' Hawley 2014-09-18 20:57:16 UTC
Adding some more info, I put the 0.73 test Debug firmware on the board and tried booting it I get (towards the end)

====================================

[...]

InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845A718
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 784573A8
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845A798
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78457028
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845A818
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78456AA8
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845A898
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78456728
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845A918
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 784563A8
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845A998
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78456028
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845AA18
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78455AA8
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845AA98
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78455728
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845AB18
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 784553A8
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845AB98
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78455028
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845AC18
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78454AA8
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7845AC98
InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A 78454728
FvbProtocolWrite: Lba: 0x0 Offset: 0x31A6 NumBytes: 0x1, Buffer: 0x7B233860
FvbProtocolWrite: Lba: 0x0 Offset: 0x3334 NumBytes: 0x3C, Buffer: 0x7B317010
FvbProtocolWrite: Lba: 0x0 Offset: 0x3336 NumBytes: 0x1, Buffer: 0x7B317012
FvbProtocolWrite: Lba: 0x0 Offset: 0x3370 NumBytes: 0x86, Buffer: 0x7B31704C
FvbProtocolWrite: Lba: 0x0 Offset: 0x3336 NumBytes: 0x1, Buffer: 0x7B317012
FvbProtocolWrite: Lba: 0x0 Offset: 0x31A6 NumBytes: 0x1, Buffer: 0x7B233860
XhcCreateUsb3Hc: Capability length 0x80
XhcCreateUsb3Hc: HcSParams1 0x7000820
XhcCreateUsb3Hc: HcSParams2 0x84000054
XhcCreateUsb3Hc: HcCParams 0x200077C1
XhcCreateUsb3Hc: DBOff 0x3000
XhcCreateUsb3Hc: RTSOff 0x2000
XhcCreateUsb3Hc: UsbLegSupOffset 0x460
XhcCreateUsb3Hc: DebugCapSupOffset 0x480
XhcSetBiosOwnership: called to set BIOS ownership
XhcResetHC!
XhcInitSched:DCBAA=0x783E0000
XhcInitSched:XHC_CRCR=0x783E0140
XhcInitSched:XHC_EVENTRING=0x783E1140
InstallProtocolInterface: 3E745226-9818-45B6-A2AC-D7CD0E8BA2BC 783F0038
XhcDriverBindingStart: XHCI started for controller @ 78450418
XhcResetHC!
XhcInitSched:DCBAA=0x783E0000
XhcInitSched:XHC_CRCR=0x783E0140
XhcInitSched:XHC_EVENTRING=0x783E1140
XhcReset: status Success
XhcGetState: current state 0
XhcSetState: status Success
InstallProtocolInterface: 240612B7-A063-11D4-9A3A-0090273FC14D 783CE020
XhcGetCapability: 7 ports, 64 bit 1
UsbRootHubInit: root hub 783CDF18 - max speed 3, 7 ports
XhcClearRootHubPortFeature: status Success
UsbEnumeratePort: port 0 state - 201, change - 01 on 783CDF18
UsbEnumeratePort: Device Connect/Disconnect Normally
UsbEnumeratePort: new device connected at port 0
XhcUsbPortReset!
XhcSetRootHubPortFeature: status Success
XhcClearRootHubPortFeature: status Success
XhcClearRootHubPortFeature: status Success
Enable Slot Successfully, The Slot ID = 0x1
    Address 1 assigned successfully
UsbEnumerateNewDev: hub port 0 is reset
UsbEnumerateNewDev: device is of 1 speed
UsbEnumerateNewDev: device uses translator (0, 0)
UsbEnumerateNewDev: device is now ADDRESSED at 1
UsbEnumerateNewDev: max packet size for EP 0 is 8
Evaluate context
UsbBuildDescTable: device has 1 configures
UsbGetOneConfig: total length is 59
UsbParseConfigDesc: config 1 has 2 interfaces
UsbParseInterfaceDesc: interface 0(setting 0) has 1 endpoints
UsbParseInterfaceDesc: interface 1(setting 0) has 1 endpoints
Configure Endpoint
UsbEnumerateNewDev: device 1 is now in CONFIGED state
UsbSelectConfig: config 1 selected for device 1
UsbSelectSetting: setting 0 selected for interface 0
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 783CCC98
InstallProtocolInterface: 2B2F68D6-0CD2-44CF-8E8B-BBA20B1B5B75 783CDC40
UsbConnectDriver: TPL before connect is 8, 783CCD18
InstallProtocolInterface: 387477C1-69C7-11D2-8E39-00A0C969723B 783CBAB8
InstallProtocolInterface: DD9E7534-7762-4698-8C14-F58517A625AA 783CBAD0
InstallProtocolInterface: D3B36F2B-D551-11D4-9A46-0090273FC14D 0
UsbConnectDriver: TPL after connect is 8
UsbSelectSetting: setting 0 selected for interface 1
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 783C9E18
InstallProtocolInterface: 2B2F68D6-0CD2-44CF-8E8B-BBA20B1B5B75 783CAE40
XhcClearRootHubPortFeature: status Success
UsbEnumeratePort: port 1 state - 01, change - 01 on 783CDF18
UsbEnumeratePort: Device Connect/Disconnect Normally
UsbEnumeratePort: new device connected at port 1
XhcUsbPortReset!
XhcSetRootHubPortFeature: status Success
XhcClearRootHubPortFeature: status Success
XhcClearRootHubPortFeature: status Success
Enable Slot Successfully, The Slot ID = 0x2
    Address 2 assigned successfully
UsbEnumerateNewDev: hub port 1 is reset
UsbEnumerateNewDev: device is of 2 speed
UsbEnumerateNewDev: device uses translator (0, 0)
UsbEnumerateNewDev: device is now ADDRESSED at 2
UsbEnumerateNewDev: max packet size for EP 0 is 64
Evaluate context
UsbBuildDescTable: device has 1 configures
UsbGetOneConfig: total length is 39
UsbParseConfigDesc: config 1 has 1 interfaces
UsbParseInterfaceDesc: interface 0(setting 0) has 3 endpoints
XhcCheckUrbResult: BABBLE_ERROR! Completecode = 3
XhcControlTransfer: error - Not Ready, transfer - 40
UsbBuildDescTable: get language ID table Unsupported

====================================

I get the BABBLE_ERROR twice, after it gets to the point mentioned above, it seems to clear the screen and do another probe, afterwards I get the same thing, it stalls a bit then continues and actually boots.

Using the 0.73 Release firmware the board does, eventually, boot (now that I've seen it boot with the debug firmware), it just takes significantly longer than is expected with no output, either on the serial port or the screen indicating what's going on.
Comment 2 He, Tim 2014-09-25 07:57:52 UTC
I'll try to reproduce the issue, then investigate it. Thanks John.
Comment 3 Mang Guo 2014-11-14 07:44:07 UTC
I have been able to reproduce the problem with SMC USB2.0 to Ethernet Adapter, and I will continue to investigate. Thanks.
Comment 4 Henry Bruce 2015-01-15 20:15:24 UTC
Saw similar behavior with Rosewill RNX-N18UBE USB Wifi adaptor.
lsusb shows
ID 0bda:8172 Realtek Semiconductor Corp. RTL8191SU 802.11n WLAN Adapter

Hardware is A2. Saw same behavior with 0.71, 0.73 and pre-release firmware.

This is a blocker for my project.
Comment 5 John 'Warthog9' Hawley 2015-02-10 00:15:04 UTC
Still seeing this on 0.77, Mang do you have any update on this?
Comment 6 Henry Bruce 2015-02-18 21:39:24 UTC
Behavior is worse in fw 0.77. Now boot is delayed for 2 minutes and USB stack is dead when EFI prompt is displayed on HDMI output (i.e. USB mouse and keyboard are dead).

I reverted to v0.76 and saw original behavior (30s boot delay).

The Realtek USB adapter driver crashes on OS boot for Poky v1.6, so boot delay is prolonging debug time. Please raise priority of this bug.
Comment 7 David_Wei 2015-02-26 02:45:51 UTC
USB host controller drivers and device drivers are updated by V0.77. V0.76 drivers are from UDK2014 (open source release of UEFI BIOS core ), while V0.77 are from UDK2014.SP1 (open source release of UEFI core ). I will inverstigate it. Meanwhile, you could download those drivers' source code from EDKII open source SVN when doing inverstigtion from your side.
Comment 8 Henry Bruce 2015-02-26 23:14:30 UTC
I got an engineering build of v0.78 from John Hawley and behavior has reverted to 30s delay as seen with 0.76.
Comment 9 Darren Hart 2015-02-27 03:59:25 UTC
(In reply to comment #8)
> I got an engineering build of v0.78 from John Hawley and behavior has
> reverted to 30s delay as seen with 0.76.

While I'm glad to see we don't take 2 minutes to boot, I suggest we leave this as Open and Medium+ to eliminate the remaining delay due to these devices.
Comment 10 David_Wei 2015-03-02 09:35:25 UTC
Debug Progress: 
This long delay is introduced by UEFI XHCI driver (open source code). This driver fails to properly handle babble error of USB control transfer and fails to recover driver itself or device (not sure which one, need root cause) from error condition. After the babble error, the USB ethernet adapter's control endpoint falls into "halt" state and does not respond to any host requests. But driver keeps waiting the device to repsond until time out. A temp solution of recovering from error state has been ready, but need to talk with this driver's owner to figure out a root cause.
Comment 11 Henry Bruce 2015-03-02 17:32:13 UTC
Realtek driver code can be browsed at https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/staging/rtl8712?id=refs/tags/v3.14.34

TODO file suggests contacting Larry Finger <Larry.Finger@lwfinger.net> and
Florian Schilhabel <florian.c.schilhabel@googlemail.com>.
Comment 12 David_Wei 2015-03-03 05:36:28 UTC
Figured out solution with USB expert. Is updating UEFI's XHCI driver.

(1)Babble error puts endpoint into "Halt" state: 

XHCI spec: 4.10.2.4 Babble Detected Error
When a device transmits more data on the USB than the host controller is expecting for a transaction, it is defined to be babbling. In general, this is called a Babble Error. When a device sends more data than 
the TD Transfer Sizebytes (TD Babble), unexpected activity that persists beyond a specified point in a (micro)frame (Frame Babble), or a packet greater than Max Packet Size (Packet Babble), the host controller shall set the Babble Detected Errorin the Completion Codefield of the TRB, generate an Error 
Event, and halt the endpoint. 


(2) How to recover an endpoint form "Halt" state:
XHCI SPEC section 4.10.2.1:
System software shall use a Reset Endpoint Command(section 4.11.4.7) to remove the Halted condition in the xHC. After the successful completion of the Reset Endpoint Command, the Endpoint Context is transitioned from the Haltedto the Stoppedstate and the Transfer Ring of the endpoint is 
reenabled. The next write to the Doorbell of the Endpoint will transition the Endpoint Context from the Stopped to the Runningstate.

Note: The Reset Endpoint Commandfor the endpoint shall complete successfully andthe halt condition on the USB device shall be successfully cleared before attempting to restart the Transfer Ring by 
ringing its doorbell
Comment 13 Darren Hart 2015-03-03 05:48:13 UTC
(In reply to comment #12)
> Figured out solution with USB expert. Is updating UEFI's XHCI driver.
> 
> (1)Babble error puts endpoint into "Halt" state: 
> 
> XHCI spec: 4.10.2.4 Babble Detected Error
> When a device transmits more data on the USB than the host controller is
> expecting for a transaction, it is defined to be babbling. In general, this
> is called a Babble Error. When a device sends more data than 
> the TD Transfer Sizebytes (TD Babble), unexpected activity that persists
> beyond a specified point in a (micro)frame (Frame Babble), or a packet
> greater than Max Packet Size (Packet Babble), the host controller shall set
> the Babble Detected Errorin the Completion Codefield of the TRB, generate an
> Error 
> Event, and halt the endpoint. 
> 
> 
> (2) How to recover an endpoint form "Halt" state:
> XHCI SPEC section 4.10.2.1:
> System software shall use a Reset Endpoint Command(section 4.11.4.7) to
> remove the Halted condition in the xHC. After the successful completion of
> the Reset Endpoint Command, the Endpoint Context is transitioned from the
> Haltedto the Stoppedstate and the Transfer Ring of the endpoint is 
> reenabled. The next write to the Doorbell of the Endpoint will transition
> the Endpoint Context from the Stopped to the Runningstate.
> 
> Note: The Reset Endpoint Commandfor the endpoint shall complete successfully
> andthe halt condition on the USB device shall be successfully cleared before
> attempting to restart the Transfer Ring by 
> ringing its doorbell

Great work David, thank you! To be clear, even if the device continues to misbehave, the system can recover without lengthy delays and continue to boot?

Is there a possibility to end up in some kind of recovery babble loop? (Yes, I totally made that up). :-)
Comment 14 David_Wei 2015-03-03 06:44:58 UTC
Hi Darren,

Q: Is there a possibility to end up in some kind of recovery babble loop? 
A: The whole UEFI USB driver stack includes XHCI host controller driver, USB bus driver and USB device driver. The babble error is detected and reported by the  XHCI host controller driver, which is at the bottom of the USB driver stack. After the error detection and recovery, XHCI host controller driver does not initial another retry. It is up to higer level driver (USB bus driver or USB device river) to determine if a retry is needed or not. 

For this bug, the babble error is reported when USB bus driver try to get Language ID string descriptor (which is an optional descriptor) of the USB network adaptor. When an error detected, USB bus driver "knows" the Language ID string descriptor does not exists or some bad happened. It then just reports Language ID string descriptor as NULL and does not continue retrying.
Comment 15 David_Wei 2015-03-03 09:25:59 UTC
Created attachment 2428 [details]
Enhanced error handling of XHCI control transfer

Please help test attached two images (debug and release version) with your USB network adapeter to see if the delay has gone.
Comment 16 Henry Bruce 2015-03-03 17:33:05 UTC
I installed MNW2MAX1.X64.0078.R01.1503031714.bin and it has removed the 30s delay on EFI boot and Realtek Wi-Fi driver is working on Poky Linux. A good result all round. Nice work. When can I expect a release with this fix included?
Comment 17 Darren Hart 2015-03-03 18:23:51 UTC
David, Henry, thank you both, great news.
Comment 18 Mike Wu 2015-03-04 01:51:38 UTC
The fix will be in 0.78
Comment 19 David_Wei 2015-03-24 00:54:11 UTC
Assgin to John. V0.78 release has fixed ths issue. Please check it.
Comment 20 John 'Warthog9' Hawley 2015-03-31 23:14:59 UTC
Fixed with the 0.78 release