Bug 6141

Summary: Rootfs ext3 filesystem was corrupt
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Adrien Destugues <pulkomandy>
Component: coreAssignee: Chen Qi <Qi.Chen>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: bluelightning, meta.mr.watcher, meta.watcher, poky.bs.watcher, poky.watcher, richard.purdie, sgw
Version: 1.5.1   
Target Milestone: 1.5.3   
Hardware: Other   
OS: arm   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
local.conf used when the bug occured.
none
do_rootfs log file none

Description Adrien Destugues 2014-04-09 17:58:49 UTC
Hi,
We are building a system using Yocto and deriving Poky (dora version). Things were working fine on http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dora&id=1a4fd0dd66ee4f1acbbe00f32085bf0d20530526 , but then we updated to http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dora&id=f1276b066223e7f501f7f711680215ff8edee252 . Starting from there, we would get a corrupted ext3 image of the rootfs (a tar.gz image would still be fine and we can use it to boot over NFS).

We got a similar issue on two different build machines. This persisted until we added RM_OLD_IMAGE="1" to local.conf, which fixed the issue. Before that, we tried to rebuild the image ( bitbake -f -c clean image ; bitbake image ) without success, still getting a corrupted output.

The corrupted filesystem would make fsck raise some warnings. It lacked a /etc dir, had various binaries in / instead of the expected /usr/bin, and of course didn't boot as expected.

Could it be that the previous image was reused somehow, leading to a mix of the two different ext3 block sizes? I'd expect the build to start fresh at each run. I don't see any other commit in the range we merged that would trigger this issue.
Comment 1 Saul Wold 2014-05-13 16:57:07 UTC
I tried to reproduce this issue and was not able to, when I did a clean build starting at the first commit you mentioned and then built a second time with the second commit which changes the inode size, I had 2 distinct .ext3 filesystems built.

Do you have any additional settings in your local.conf that might be helpful to know about?  Can you attach it here.
Comment 2 Adrien Destugues 2014-05-14 15:17:20 UTC
Hi,

We had this issue again, this time without any update to Yocto, so it seems this is not related to the block size change after all. I'm not sure any of the settings in local.conf are relevant, but I'll attach it anyway.

One thing that could be relevant is we are building our image with IMAGE_ROOTFS_EXTRA_SPACE="50000". It was suggested we may be running out of inode numbers in the image, but I don't know if that's related to the image size in any way.
Comment 3 Adrien Destugues 2014-05-14 15:18:40 UTC
Created attachment 1956 [details]
local.conf used when the bug occured.
Comment 4 Richard Purdie 2014-05-14 16:58:12 UTC
Do you have a copy of the rootfs log from a failed image? Are there any errors or warnings reported there?
Comment 5 Richard Purdie 2014-05-14 17:02:09 UTC
Lowering priority since this doesn't seem as widespread as we first worried about (1.5.2 didn't run into it during QA)
Comment 6 Adrien Destugues 2014-06-11 15:30:58 UTC
Created attachment 1998 [details]
do_rootfs log file

Hi,
Sorry for the delay.
Here is the log from do_rootfs.

Starting from line 93148, there is a "could not allocate block in ext2 filesystem" for every operation.

Running out of space is ok, but it would be nice to error out in that case. I guess the inodes and other FS info are taking more space than expected (because there are many small files) and we need to add more free space for this to work reliably.
Comment 7 Stephen K Jolley 2014-06-18 17:44:20 UTC
The Info appears to have been supplied, moving out of NEEDINFO until another question is asked per RP.
Comment 8 Chen Qi 2014-07-02 03:02:55 UTC
Module: openembedded-core.git
Branch: dora
Commit: 468d3e60ee10348578f78f846e87c02359fdb8bf
URL:    http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=468d3e60ee10348578f78f846e87c02359fdb8bf

Author: Chen Qi <Qi.Chen@windriver.com>
Date:   Thu Jun 19 10:11:33 2014 +0800

populate-extfs.sh: error out if debugfs encounters some error

Previously, even if we encounter some error when populating the
ext filesystem, we don't error out and the rootfs process still
succeeds.

However, what's really expected is that the populate-extfs.sh script
should error out if something wrong happens when using `debugfs' to
generate the ext filesystem. For example, if there's not enough block
in the filesystem, and allocating a block for some file fails, the
failure should not be ignored. Otherwise, we will have a successful
build but a corrupted filesystem.

The debugfs returns 0 as long as the command is valid. That is, even
if the command fails, the debugfs still returns 0. That's really a
pain here. That's why this patch checks the error output to see whether
there's any error logged.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>