Bug 5639

Summary: NFS: directory usr/lib contains a readdir loop
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Tudor Florea <tudor.florea>
Component: connectivityAssignee: Jason Wessel <jason.wessel>
Status: VERIFIED FIXED QA Contact: Andra Necula <andrax.g.necula>
Severity: major    
Priority: Medium+ CC: alexandru.c.georgescu, andrax.g.necula, bluelightning, cristian.iorga, jason.wessel, mark.hatle, meta.mr.watcher, meta.watcher, Qi.Chen, richard.purdie, sgw, yi.zhao
Version: 1.5   
Target Milestone: 1.6   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Tudor Florea 2013-12-06 12:52:17 UTC
"ls" command on different directories with many files (e.g. /usr/lib/) produce the following output on console:

NFS: directory usr/lib contains a readdir loop.Please contact your server vendor.  The file: audit has duplicate cookie 421165640
Some files are is not listed. Some files are duplicated.


 
Reproduced on custom image based on:
Build Configuration:
BB_VERSION        = "1.21.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-12.10"
TARGET_SYS        = "i586-poky-linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "1.5+snapshot-20131206"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              
meta-yocto        
meta-yocto-bsp    = "master:a17ffbdbb7d142a94ef842a32b749b18e2e7f0a1"


runqemu-extract-sdk tmp/deploy/images/qemuppc/<custom-image>-qemuppc.tar.gz tmp/deploy/images/qemuppc/rootfs/
runqemu tmp/deploy/images/qemuppc/vmlinux-qemuppc.bin tmp/deploy/images/qemuppc/rootfs/

This happens on HW targets also.

This seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=739222 which was closed as duplicated of restriceted case https://bugzilla.redhat.com/show_bug.cgi?id=813070
Comment 1 Saul Wold 2013-12-10 23:26:21 UTC
This seems to be an issue in Dora also, this is setup from the unfs-server-native, it might even be an older problem.
Comment 2 Alexandru Georgescu 2013-12-11 08:41:20 UTC
Tudor, on what arches have you tried?
I see that your conf is specifying the qemux86 but your rootfs is qemuppc.

I have tried with a qemux86-64 built image and didn't see this issue.

commit c51730aaa91066cfb51ad6ec2920fb4f9d9ea94c
Comment 3 Yi Zhao 2013-12-11 10:02:45 UTC
I also can not reproduce this issue on qemuarm, qemumips, and qemuppc with 1.6 M1 rc1 build. (git rev: master/c51730aaa91066cfb51ad6ec2920fb4f9d9ea94c)
Comment 4 Tudor Florea 2013-12-11 10:05:23 UTC
Alexandru,
I don;t think the arch is relevant.
I reporduced now the issue on vanilla core-image-sato on powerpc:

bitbake core-image-sato

Build Configuration:
BB_VERSION        = "1.21.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-12.10"
TARGET_SYS        = "powerpc-poky-linux"
MACHINE           = "qemuppc"
DISTRO            = "poky"
DISTRO_VERSION    = "1.5+snapshot-20131211"
TUNE_FEATURES     = "m32 fpu-hard ppc7400 altivec"
TARGET_FPU        = ""
meta              
meta-yocto        
meta-yocto-bsp    = "master:a17ffbdbb7d142a94ef842a32b749b18e2e7f0a1"

bitbake meta-ide-support

runqemu-extract-sdk tmp/deploy/images/qemuppc/core-image-sato-qemuppc.tar.bz2 tmp/deploy/images/qemuppc/rootfs

runqemu tmp/deploy/images/qemuppc/vmlinux-qemuppc.bin tmp/deploy/images/qemuppc/rootfs
Comment 5 Alexandru Georgescu 2013-12-11 15:23:46 UTC
Okay, managed to reproduce:

root@qemux86-64:~/standalone-automated-MAR-4/sudoku-savant-1.3# ls /usr/lib/pkNFS: directory usr/lib contains a readdir loop.Please contact your server vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
NFS: directory usr/lib contains a readdir loop.Please contact your server vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
oNFS: directory usr/lib contains a readdir loop.Please contact your server vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
NFS: directory usr/lib contains a readdir loop.Please contact your server vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
^C
onfig/kdNFS: directory lib/pkgconfig contains a readdir loop.Please contact your server vendor.  The file: xcb-screensaver.pc has duplicate cookie 4106177285
NFS: directory lib/pkgconfig contains a readdir loop.Please contact your server vendor.  The file: xcb-screensaver.pc has duplicate cookie 4106177285

Seems that when pressing tab for autocomplete goes crazy.
Comment 6 Alexandru Georgescu 2013-12-11 15:51:00 UTC
(In reply to comment #5)
> Okay, managed to reproduce:
> 
> root@qemux86-64:~/standalone-automated-MAR-4/sudoku-savant-1.3# ls
> /usr/lib/pkNFS: directory usr/lib contains a readdir loop.Please contact
> your server vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
> NFS: directory usr/lib contains a readdir loop.Please contact your server
> vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
> oNFS: directory usr/lib contains a readdir loop.Please contact your server
> vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
> NFS: directory usr/lib contains a readdir loop.Please contact your server
> vendor.  The file: xml2Conf.sh has duplicate cookie 1167227051
> ^C
> onfig/kdNFS: directory lib/pkgconfig contains a readdir loop.Please contact
> your server vendor.  The file: xcb-screensaver.pc has duplicate cookie
> 4106177285
> NFS: directory lib/pkgconfig contains a readdir loop.Please contact your
> server vendor.  The file: xcb-screensaver.pc has duplicate cookie 4106177285
> 
> Seems that when pressing tab for autocomplete goes crazy.

Testing the two versions (for qemux86 and qemux86-64) got me to paste the log from qemux86-64 in this qemux86 machine. I can confirm that the same type of issue happens on both arches.
Comment 7 Alexandru Georgescu 2013-12-11 15:52:31 UTC
*** Bug 5638 has been marked as a duplicate of this bug. ***
Comment 8 Alexandru Georgescu 2013-12-11 15:54:19 UTC
*** Bug 5637 has been marked as a duplicate of this bug. ***
Comment 9 Saul Wold 2013-12-11 16:20:52 UTC
Yi, what is your host machine OS and which image did you test with?  Please 
test with -sato or -sato-sdk, I wonder if it's image size dependant.

Alex: What host OSes did you test with?
Comment 10 Alexandru Georgescu 2013-12-11 16:47:01 UTC
(In reply to comment #9)
> Yi, what is your host machine OS and which image did you test with?  Please 
> test with -sato or -sato-sdk, I wonder if it's image size dependant.
> 
> Alex: What host OSes did you test with?

Ubuntu 13.10 64bit.

I will try now with f19 64bit to see if this happens there too. I tested with sato-sdk.
Comment 11 Alexandru Georgescu 2013-12-11 17:15:52 UTC
The bug reproduces in FC19 64bit as well.
Comment 12 Alexandru Georgescu 2013-12-11 17:17:35 UTC
(In reply to comment #3)
> I also can not reproduce this issue on qemuarm, qemumips, and qemuppc with
> 1.6 M1 rc1 build. (git rev: master/c51730aaa91066cfb51ad6ec2920fb4f9d9ea94c)

Yi, try to reproduce it by booting up the qemu with serial. The error should show in the serial console!

Thanks!
Comment 13 Alexandru Georgescu 2013-12-11 17:25:50 UTC
Saul,
this happens for Sato only image as well!
Comment 14 Tudor Florea 2013-12-11 17:30:00 UTC
Reproduced the bug on qemux86, core-image-sato as well.
The issue has to do with the number of file is one directory exported rather than the size of the image:
root@qemux86:/tmp1# ls -1 | wc -l
79
root@qemux86:/tmp1# touch 80
root@qemux86:/tmp1# ls -1 | wc -l
80
root@qemux86:/tmp1# touch 81
root@qemux86:/tmp1# ls -1 | wc -l
203
Comment 15 Tudor Florea 2013-12-11 17:31:39 UTC
the error pops up on the console only.
However one may see the issue using dmesg
Comment 16 Saul Wold 2013-12-12 21:56:12 UTC
This is clearly related to our unfs_server, which is quite old and this problem 
may have been around for a while and we just have not hit it directly like 
this.  It seems to be a know issue, there are a couple of other userspace nfs 
servers around that I am looking at as alternatives.
Comment 17 Jason Wessel 2013-12-12 22:42:34 UTC
In case you are actually curious what causes this problem it has to do with the 64 bit ext3 / ext4 fs's.  Modern kernels use a 64 bit hash for readdir / telldir, and nfsv2 cannot support this, becuase the inode hash is 32 bit.  The directory loops and such are a direct product of the hash being the same for two different directories. 

The solution used in Wind River Linux was to use unfs3.
Comment 18 Saul Wold 2014-02-27 15:41:21 UTC
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=162dd389301ce238b5003d6b4f6031256532508b

Addition of UNFS3 and supporting changes to qemu resolves this issue
Comment 19 Andra Necula 2014-07-28 15:34:26 UTC
Verified on branch 'origin/master' commit addeef1dc91306175824b1f5587e40a22773cff4