2881 2012-08-02 18:23:41 +0000 Under pseudo rm of files > 2GB fail with errno 75 = Value too large for defined data type 2013-05-01 15:43:32 +0000 1 1 7 Build System, Metadata & Runtime Meta-yocto meta-yocto unspecified x86 Multiple RESOLVED FIXED Medium normal 1.4 1 andrei seebs jessica.zhang poky.bs.watcher poky.watcher sgw --- --- --- oldest_to_newest 23759 0 andrei 2012-08-02 18:23:41 +0000 cannot remove ‘rpi-hwup-image-raspberrypi-20120802151428.rootfs.rpi-sdimg.bak’: Value too large for defined data type 23761 1 andrei 2012-08-02 19:56:58 +0000 Fix: http://lists.linuxtogo.org/pipermail/openembedded-core/2012-August/027186.html 23766 2 seebs 2012-08-02 23:35:30 +0000 Okay, I've done some poking at this, and I think I have a fix. In fact, I would go so far as to say that I've tested the fix, a little. Only on a 64-bit box, but I made a 32-bit pseudo and a 32-bit app that called rmdir, and I did get the behavior I expected. So that's the PSEUDO_1_4_1 branch, if someone wants to give it a whirl. I'll get it code reviewed and then see about trying to update. 23785 3 andrei 2012-08-03 08:36:18 +0000 Your patched seemed OK - obviously much better than my 2 line patch. :) But while testing... tar: ./var/volatile/cache/ldconfig/aux-cache: Unknown file type; file ignored So now it crashes in tar... ag 23819 4 andrei 2012-08-06 10:07:43 +0000 More info: Ubuntu says: --- The crashed program seems to use third-party or local libraries: /home/agherzan/work/personal/yocto/2012-07-24-rpi/tmp/sysroots/i686-linux/usr/lib/pseudo/lib/libpseudo.so It is highly recommended to check if the problem persists without those first. Do you want to continue the report process anyway? --- (In reply to comment #3) > Your patched seemed OK - obviously much better than my 2 line patch. :) > > But while testing... > tar: ./var/volatile/cache/ldconfig/aux-cache: Unknown file type; file ignored > So now it crashes in tar... > > ag 23830 5 seebs 2012-08-06 17:39:51 +0000 Okay, I don't have a 32-bit system handy (wow, never thought I'd say THAT). I'd be interested in more data on this; in particular, in stat information for the file, both inside and outside pseudo. Worst case... ugh. I wish I'd gotten to fixing the debug output yet. I want to know what tar's calling that it's getting an unexpected response from, but the only way I know of to get real information about wrappers called is PSEUDO_DEBUG=4, which is ridiculously verbose. 23832 6 seebs 2012-08-06 18:24:31 +0000 Okay, I looked into tar a bit. Ultimately, the value in st_mode (which is the only thing that can trigger that diagnostic, unless you're using a V7_FORMAT archive) comes from a call to either stat() or lstat(). So the call chain from those is roughly: stat calls fxstatat which does a real fxstatat() on the buffer, then calls the fxstatat64() wrapper on a 64-bit buffer, and overwrites all the standard/known stat buffer values in the plain stat from the stat64. This could result in a meaningless size (which you'd have gotten anyway), but it should not result in an invalid mode; the mode should either come from the filesystem or the database. If you can easily arrange to replace the version of tar being used, could you run with one that dumps the mode it reports, and/or run the tar command with PSEUDO_DEBUG=2 and get me a copy of the logs? (Warning: they will be huge. You might want to do a dummy command like "tar cf /dev/null var/blah/blah/that-single-file" to shorten them.) 23833 7 andrei 2012-08-06 18:45:03 +0000 Here is an example with DEBUG=4 25313: called: __fxstat64 25313: fstat /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-onetouch.ko [fd 14] (+buf) [dev/ino: 2054/16781491] (0100644): processing request [ino 16781491] 25313: sending request [ino 16781491] 25313: sending a message: ino 16781491 25313: msg type 3 (fstat), external path /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-onetouch.ko, mode 0100644 25313: got header, type 4, pathlen 0 25313: got response type 4 25313: (25313) fail mode 0100644 uid 1000:1000 25313: completed: __fxstat64 (errno: 0) 25313: called: close 25313: close /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-onetouch.ko [fd 14]: processing request [ino 0] 25313: (25313) (no request) 25313: completed: close (errno: 0) 25313: called: __fxstatat64 25313: base_path: /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage</>ums-sddr09.ko 25313: root_path [__fxstatat64, 280]: '/home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko' from 'ums-sddr09.ko' 25313: statat /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko (+buf) (0100644): processing request [ino 16781968] 25313: sending request [ino 16781968] 25313: sending a message: ino 16781968 25313: msg type 3 (stat), external path /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko, mode 0100644 25313: got header, type 4, pathlen 0 25313: got response type 4 25313: (25313) succeed mode 01143570755 uid 0:0 25313: completed: __fxstatat64 (errno: 0) 33188 Before error ./lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko This is just one file reported with error. There are a lot of them! 23835 8 andrei 2012-08-06 18:59:15 +0000 In pseudo: 1140670644 device_table Before error ./etc/device_table Outside pseudo: 100755 device_table 23899 9 seebs 2012-08-08 23:43:14 +0000 Okay, this one's turning interesting; I got access to a proper 32-bit Linux machine (Fedora 17, although it shouldn't matter), and I can't actually get it to fail. Hmm. Of some interest: I note that it seems as though the last three digits are a plausible mode, but the rest is all garbled. Hmm. I'll keep looking. Can you trigger this outside the build system so the reproducer is a little smaller? If you could get this to happen on a smaller test case, I could probably look at the files.db and maybe figure something out. 23900 10 seebs 2012-08-09 00:13:35 +0000 I should add some explanation of those logs. The key point: 25313: statat /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko (+buf) (0100644): processing request [ino 16781968] 25313: sending request [ino 16781968] 25313: sending a message: ino 16781968 25313: msg type 3 (stat), external path /home/agherzan/work/personal/yocto/2012-08-06-rpi/tmp/work/raspberrypi-poky-linux-gnueabi/rpi-hwup-image-1.0-r0/rootfs/lib/modules/3.1.9/kernel/drivers/usb/storage/ums-sddr09.ko, mode 0100644 25313: got header, type 4, pathlen 0 25313: got response type 4 25313: (25313) succeed mode 01143570755 uid 0:0 25313: completed: __fxstatat64 (errno: 0) Which is to say: We got an __fxstatat() request for ums-sddr09.ko, which had an existing mode of 0100644. We sent the request to the server, and the response had a new mode of 01143570755. Which is, it turns out, probably not correct. There are two possibilities I can think of: 1. Something in the sequence by which the response gets back from the server is busted. 2. The pseudo database itself is full of really strange bogus values. 2a. Which got in through some other part of this being screwed up. 2b. Because the files are corrupt for some unrelated reason. I guess the first thing would be to see whether, in a new build directory, you see similar problems. I was pretty sure this couldn't break existing stuff, and wouldn't change the bits sent over the wire, but it is possible that I missed something somewhere. 23925 11 seebs 2012-08-09 20:32:32 +0000 Well, whaddya know. Ran a whole build on the 32-bit box: tar: ./var/volatile/cache/ldconfig/aux-cache: Unknown file type; file ignored I should be able to track this down now. 23926 12 seebs 2012-08-09 23:33:13 +0000 Okay, found it. There's a new patch on the PSEUDO_1_4_1 branch (and master) to fix that. What's wrong is that I used unwrapped stat() calls, and that can fail inside a chroot. The problem is, in the case where something creates a file under another name, then moves it, and is inside a chroot, we end up creating a link using a statbuf which is stack garbage. The reason it's hard to reproduce is partially the need to be in a chroot for it to go wrong, and partially that sometimes the garbage didn't happen to have an invalid mode. 24128 13 andrei 2012-08-16 22:08:34 +0000 Thanks for the work done in this matter. Now my problem is solved. So please submit a an update to pseudo for this. Thanks, ag 32797 14 sgw 2013-04-30 23:11:26 +0000 can this be closed? 32799 15 seebs 2013-04-30 23:19:25 +0000 I think so, it's in 1.4 and 1.5 both. 32815 16 sgw 2013-05-01 15:43:32 +0000 Resolved