For the jasperforest WRL 5.0 BSP, Wind River is requesting the following be enabled in the next (3.4) kernel: jasperforest drivers (NTB and Crystal Beach 3 DMA/DCA drivers) I added the dca and dmaengine features along with the ixgbe and igb features for romley. I could partially cover the above by adding those features to the jasperforest BSP, but it would be nice to have a better description as to what exactly the requirements are in order to avoid guesswork.
The Intel NTB drivers were missing for a long while even after the platform was initially released to the public a couple of years ago. The Linux driver enabling NTB allows interprocessor communications over the PCIe NTB (non-transparent bridging) integrated on the Jasper Forest platform. The NTB logic provides high availability by connecting two independent storage systems over PCI Express. By enabling the NTB feature, the platform allows each of the connected storage to access the other’s memory window. This access potentially support a failover scenario by allowing one system to continue operating on the shared data after the other fails. There is an Intel published whitepaper for this NTB feature.
Tom I took a swag at prio and sizing (assuming this is a config option and not patch integration or driver development). Please review.
That seems reasonable - I'll change it if it turns out to be more involved. A quick scan of the Developer's guide, whitepaper and current kernel didn't turn up much in the way of actual code, but I still need to dig in some more. There were mentions of '3 drivers' each used for different use cases, so if there's a specific use-case in mind that would also be useful information. I have a sneaking suspicion that there might be userspace here.
Apparently the drivers won't currently work for Jasperforest, so we'll defer this to 1.4.
I don't have any hardware I can test with, so need to pass it on to the maintainer.
I just check for the ntb drivers, and I do not find them in any of the upstream kernels.
Hi Nitin, Have you see this link? PCI-Express Non-Transparent Bridge Support http://lwn.net/Articles/518370/ and this: Version 4 removes the transport transmit tasklet (per Dave Miller's request). http://thread.gmane.org/gmane.linux.network/244491 Thanks -Michael
Hi Michael, I have seen the NTB patches floating on the Mailing Lists. but I was not sure which is the final set. Now I am in touch with the NTB developer Jon Mason. I will get his latest patches integrated in the Linux Yocto v3.8 kernel repo. Seems like one need special hardware setup to test the NTB. Don says he has been using on the osage motherboard for his NTB work. I am not sure if my jasper forest systems is the right one for this work. But I hear from Jon, that you WR guys have couple of the osage platforms. So may be you can help here in testing out the feature. Thanks, Nitin
Michael, I have added these branches for testing out this feature: * linux-yocto-3.8 repo branches: ** meta: http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-contrib/log/?h=ntb/meta ** standard/common-pc-64/jasperforest: http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-contrib/log/?h=ntb/standard/common-pc-64/jasperforest * meta-intel branches: http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel-contrib/log/?h=nitin/ntb Can you test this out with the NTB setup? Bruce, Can you help him out with setting the local kernel repo for his work? Thanks, Nitin
Michael, What is the timeline for this to get verified? From our emails WR is going to validate it. I can provide a BSP image to test. let me know when you are ready to test NTB. If this is needed for the YP 1.4 release, then we need to resolve this bug quickly. Thanks, Nitin
Hi Nitin, I'm working on NTB verification on Jasper forest and Trenton targets, which are connected by Montlake cards over PCIe cable. Using the patch in WRLinux 5.0.1, the virtual network interface (eth1/eth2/eth3/eth4) are not able to link up. So first of all, I need to make sure no hardware issue on the card or target board. Could you provide the image and also ntb.ko/ntb_netdev.ko to test NTB? Thanks, Yunguo (In reply to comment #10) > Michael, > What is the timeline for this to get verified? From our emails WR is going > to validate it. I can provide a BSP image to test. let me know when you are > ready to test NTB. If this is needed for the YP 1.4 release, then we need to > resolve this bug quickly. > > Thanks, > Nitin
Hi Yunguo, Thanks for update on the hardware side. Currently I have implemented the NTB feature as part of the kernel, and not as a kernel module. That's the general direction we take for BSP specific features. If really needed it can easily be converted to kernel modules. The core-sato-image is of about 500MB size, which has NTB drivers already included in the kernel. I need to find a way to share it. If you have any idea for sharing let me know. I think I can use drop box to share the 500MB image file.
Here is the link to the image file. https://www.dropbox.com/s/ze3smsizpwic70j/core-image-sato-jasperforest-20130304235013.hddimg
Hi Nitin, Thanks for the help. I've downloaded the image and tried it in our lab. Testing scenario is like this: One Trenton board and one Osage board, each with a Montlake card, connected via PCIe cable. Using the image you provided, the monitor keeps printing "ntb 0000:00:03:0: Error determining link status" on both boards. And after a while, the system will enter graphic desktop. Since the ntb is not a module, and I didn't find any newly created ethx, so how to verify NTB is working? Thanks, Yunguo
I can ping each other on two Osage boards. One board works well with existing NTB driver. On the other board, I have to apply the following changes to existing NTB driver, otherwise two boards are not able to "Link Up". --- ntb_hw.c.orig 2013-04-12 09:22:54.000000000 +0800 +++ ntb_hw.c 2013-04-12 09:23:50.000000000 +0800 @@ -537,10 +537,7 @@ static int ntb_link_status(struct ntb_de if (rc) return rc; - if (status & NTB_LINK_STATUS_ACTIVE) - link_state = NTB_LINK_UP; - else - link_state = NTB_LINK_DOWN; + link_state = NTB_LINK_UP; } ntb_link_event(ndev, link_state); I'm guessing maybe there is some problem with PCIE card on the failing board, or the board itself. It is weird that, if I swapped the two cards on the two boards, the system event can't boot. Thanks, Yunguo
Hi Yunguo, Even you are using the latest upstream driver, you still need such fix (set link status to UP unconditionally)? I suppose the driver should read the link status from some registers which usually should be updated by hardware directly, if so we can say the link status is not updated correctly by the hardware. And, you'd better double check this with Intel NTB developers. If you don't know their contact or email address, just let me know.
(In reply to comment #16) > Hi Yunguo, > > Even you are using the latest upstream driver, you still need such fix (set > link status to UP unconditionally)? I suppose the driver should read the > link status from some registers which usually should be updated by hardware > directly, if so we can say the link status is not updated correctly by the > hardware. Yes, using the latest patch for 3.8.6 provided by dave.jiang@intel.com, I still have to apply my change to force up the link. I also agree with you that, the link status should be updated by the hardware, so this change is only a workround for a failing card/board, not a real fix. And this change is needed only on one board, for the other board, the latest driver is working without any change. By the way, even with this change, ntb driver must be loaded firstly on the failing board, then the working board. If ntb driver is loaded on the working board before it is loaded on the failing board, NTB is failed to ping each other. So I'm guessing there is problem with card or board. Oak Creek Canyon board will be shipped to CDC lab soon, and I will verify NTB on it then. > > And, you'd better double check this with Intel NTB developers. If you don't > know their contact or email address, just let me know. I have let Jon Mason know this change, he couldn't tell the root cause, as Osage board is not the exact board what he was working on. Thanks, Yunguo
Hi All, I have shared a newly generated image for jasperforest BSP with NTB support here: https://www.dropbox.com/s/zrxe1da2pv9oahc/core-image-sato-jasperforest-20130416023606.hddimg This is with the latest kernel in the linux-yocto-3.8 repo, which is 3.8.4. Also I have converted the kernel drivers as modules now. If it is good enough we can also push this to YP 1.4 release. Thanks, Nitin
Nitin, where are we with this? Did it make 1.4? If not, do we just need to move to 1.5?
This is still not verified with the Yocto Project Kernel. And this is the right candidate for 1.5
Yunguo Wei, Did you get any chance to test the image on the jasperforest and NTB hardware. If there is some progress then we have chance to put this in the upcomign 1.4.1 YP release. Nitin
Hi Nitin, Sorry for the late response as we are busy with oak creek canyon bsp. I've tried the image you provided. Here are the problems I found: 1. x server is not able to load. 2. NTB is not built-in or supplied as a module. root@jasperforest:/lib# zcat /proc/config.gz |grep -i ntb # CONFIG_NTB is not set 3. USB Serial converter is not supported. As one of OCC board's built-in serial is not working, I have to use a USB2Serial converter to log in the system. Can you help with these issues? Thanks, Yunguo (In reply to comment #21) > Yunguo Wei, > > Did you get any chance to test the image on the jasperforest and NTB > hardware. If there is some progress then we have chance to put this in the > upcomign 1.4.1 YP release. > > Nitin
Hi Yunguo, Thanks for the update. I can make another image to overcome these issues. But before going there I would like to get some clarifications. After discussing this matter with Michael Ling, I am understanding the requirements bit differently now. Last time I was enabling NTB in the latest v3.8.4 Yocto kernel, while Michael mentioned this is needed for v3.4 LTSI kernel for WRL. So I will go back to enabling v3.4 Yocto kernel for NTB feature. Also as per the NTB developer Jon Mason, NTB driver is not yet fully ready for the jasperforest hardware. So to enable NTB specifically for jasperforest BSP, we will need to wait until the all the needed NTB patches for jasperforest are public. From your replies about validation, I understand you were testing these NTB changes on hardware different than the jasperforest platform. Do you have the right hardware for testing it on Jasperforest platforms? Or do you think testing on other platforms should be good enough. So we still do not have everything needed for NTB enabling in Yocto Project yet, and NTB enabling may take still more time. Nitin
Hi Nitin, Sorry, we don't have two working jasper forest hardwares available, but we do have two oak creek canyon boards. I think occ board is good to me. Michael, please correct me if anything is incorrect. Regards, Yunguo
the NTB features is not applicable to Jasperforest BSP.