| Summary: | Under pseudo rm of files > 2GB fail with errno 75 = Value too large for defined data type | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Meta-yocto | Reporter: | Andrei Gherzan <andrei> |
| Component: | meta-yocto | Assignee: | Seebs <seebs> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | jessica.zhang, poky.bs.watcher, poky.watcher, sgw |
| Version: | unspecified | ||
| Target Milestone: | 1.4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
|
Description
Andrei Gherzan
2012-08-02 18:23:41 UTC
Okay, I've done some poking at this, and I think I have a fix. In fact, I would go so far as to say that I've tested the fix, a little. Only on a 64-bit box, but I made a 32-bit pseudo and a 32-bit app that called rmdir, and I did get the behavior I expected. So that's the PSEUDO_1_4_1 branch, if someone wants to give it a whirl. I'll get it code reviewed and then see about trying to update. Your patched seemed OK - obviously much better than my 2 line patch. :) But while testing... tar: ./var/volatile/cache/ldconfig/aux-cache: Unknown file type; file ignored So now it crashes in tar... ag More info: Ubuntu says: --- The crashed program seems to use third-party or local libraries: /home/agherzan/work/personal/yocto/2012-07-24-rpi/tmp/sysroots/i686-linux/usr/lib/pseudo/lib/libpseudo.so It is highly recommended to check if the problem persists without those first. Do you want to continue the report process anyway? --- (In reply to comment #3) > Your patched seemed OK - obviously much better than my 2 line patch. :) > > But while testing... > tar: ./var/volatile/cache/ldconfig/aux-cache: Unknown file type; file ignored > So now it crashes in tar... > > ag Okay, I don't have a 32-bit system handy (wow, never thought I'd say THAT). I'd be interested in more data on this; in particular, in stat information for the file, both inside and outside pseudo. Worst case... ugh. I wish I'd gotten to fixing the debug output yet. I want to know what tar's calling that it's getting an unexpected response from, but the only way I know of to get real information about wrappers called is PSEUDO_DEBUG=4, which is ridiculously verbose. Okay, I looked into tar a bit. Ultimately, the value in st_mode (which is the only thing that can trigger that diagnostic, unless you're using a V7_FORMAT archive) comes from a call to either stat() or lstat(). So the call chain from those is roughly: stat calls fxstatat which does a real fxstatat() on the buffer, then calls the fxstatat64() wrapper on a 64-bit buffer, and overwrites all the standard/known stat buffer values in the plain stat from the stat64. This could result in a meaningless size (which you'd have gotten anyway), but it should not result in an invalid mode; the mode should either come from the filesystem or the database. If you can easily arrange to replace the version of tar being used, could you run with one that dumps the mode it reports, and/or run the tar command with PSEUDO_DEBUG=2 and get me a copy of the logs? (Warning: they will be huge. You might want to do a dummy command like "tar cf /dev/null var/blah/blah/that-single-file" to shorten them.) Here is an example with DEBUG=4 25313: called: __fxstat64 25313: fstat /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-onetouch.ko [fd 14] (+buf) [dev/ino: 2054/16781491] (0100644): processing request [ino 16781491] 25313: sending request [ino 16781491] 25313: sending a message: ino 16781491 25313: msg type 3 (fstat), external path /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-onetouch.ko, mode 0100644 25313: got header, type 4, pathlen 0 25313: got response type 4 25313: (25313) fail mode 0100644 uid 1000:1000 25313: completed: __fxstat64 (errno: 0) 25313: called: close 25313: close /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-onetouch.ko [fd 14]: processing request [ino 0] 25313: (25313) (no request) 25313: completed: close (errno: 0) 25313: called: __fxstatat64 25313: base_path: /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage</>ums-sddr09.ko 25313: root_path [__fxstatat64, 280]: '/home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko' from 'ums-sddr09.ko' 25313: statat /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko (+buf) (0100644): processing request [ino 16781968] 25313: sending request [ino 16781968] 25313: sending a message: ino 16781968 25313: msg type 3 (stat), external path /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko, mode 0100644 25313: got header, type 4, pathlen 0 25313: got response type 4 25313: (25313) succeed mode 01143570755 uid 0:0 25313: completed: __fxstatat64 (errno: 0) 33188 Before error ./lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko This is just one file reported with error. There are a lot of them! In pseudo: 1140670644 device_table Before error ./etc/device_table Outside pseudo: 100755 device_table Okay, this one's turning interesting; I got access to a proper 32-bit Linux machine (Fedora 17, although it shouldn't matter), and I can't actually get it to fail. Hmm. Of some interest: I note that it seems as though the last three digits are a plausible mode, but the rest is all garbled. Hmm. I'll keep looking. Can you trigger this outside the build system so the reproducer is a little smaller? If you could get this to happen on a smaller test case, I could probably look at the files.db and maybe figure something out. I should add some explanation of those logs. The key point: 25313: statat /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko (+buf) (0100644): processing request [ino 16781968] 25313: sending request [ino 16781968] 25313: sending a message: ino 16781968 25313: msg type 3 (stat), external path /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko, mode 0100644 25313: got header, type 4, pathlen 0 25313: got response type 4 25313: (25313) succeed mode 01143570755 uid 0:0 25313: completed: __fxstatat64 (errno: 0) Which is to say: We got an __fxstatat() request for ums-sddr09.ko, which had an existing mode of 0100644. We sent the request to the server, and the response had a new mode of 01143570755. Which is, it turns out, probably not correct. There are two possibilities I can think of: 1. Something in the sequence by which the response gets back from the server is busted. 2. The pseudo database itself is full of really strange bogus values. 2a. Which got in through some other part of this being screwed up. 2b. Because the files are corrupt for some unrelated reason. I guess the first thing would be to see whether, in a new build directory, you see similar problems. I was pretty sure this couldn't break existing stuff, and wouldn't change the bits sent over the wire, but it is possible that I missed something somewhere. Well, whaddya know. Ran a whole build on the 32-bit box: tar: ./var/volatile/cache/ldconfig/aux-cache: Unknown file type; file ignored I should be able to track this down now. Okay, found it. There's a new patch on the PSEUDO_1_4_1 branch (and master) to fix that. What's wrong is that I used unwrapped stat() calls, and that can fail inside a chroot. The problem is, in the case where something creates a file under another name, then moves it, and is inside a chroot, we end up creating a link using a statbuf which is stack garbage. The reason it's hard to reproduce is partially the need to be in a chroot for it to go wrong, and partially that sometimes the garbage didn't happen to have an invalid mode. Thanks for the work done in this matter. Now my problem is solved. So please submit a an update to pseudo for this. Thanks, ag can this be closed? I think so, it's in 1.4 and 1.5 both. Resolved |