Bug 8544 - mkfs.ext4: corrupt image for hardlinks
Summary: mkfs.ext4: corrupt image for hardlinks
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 2.1
Hardware: x86 Multiple
: High major
Target Milestone: 2.0
Assignee: Robert Yang
QA Contact: Alexandru Georgescu
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-10-20 04:35 UTC by Patrick Ohly
Modified: 2015-10-29 15:29 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Ohly 2015-10-20 04:35:15 UTC
When the rootfs contains hardlinks (example: /sbin/fsck.ext4 and its aliases), then a fsck.ext4 on the device fails, complaining for example about:

...
Oct 19 19:05:36 intel-corei7-64 systemd-fsck[672]: /dev/mmcblk0p2: Setting filetype for entry 'fsck.ext3' in /sbin (20025) to 1.
Oct 19 19:05:36 intel-corei7-64 systemd-fsck[672]: /dev/mmcblk0p2: Setting filetype for entry 'fsck.ext2' in /sbin (20025) to 1.
...
Oct 19 19:05:36 intel-corei7-64 systemd-fsck[672]: /dev/mmcblk0p2: Failed to optimize directory ??? (17): Corrupt extent header.
Oct 19 19:05:36 intel-corei7-64 systemd-fsck[672]: /dev/mmcblk0p2: ***** REBOOT LINUX *****
Oct 19 19:05:36 intel-corei7-64 systemd-fsck[672]: /dev/mmcblk0p2: 20655/368832 files (0.1% non-contiguous), 221840/....

More often than not, the rootfs partition is completely unusable afterwards.

This is based on a distro using OE-core 4038970f8ce27ac0d7a0afe2cdaa9a65108dfff5 (slighly older than current jethro branch). I've not tried to reproduce it yet with genuine OE-core or poky, but I suspect it is affected the same way.
Comment 1 Patrick Ohly 2015-10-20 06:01:08 UTC
Here's an incomplete workaround for the problem which can be used in image recipes:

# A workaround for a deficiency in OE-core's mkfs.ext4:
# it does not handle hard links correctly, generating a
# filesystem that fsck.ext4 cannot fix and instead completely
# corrupts (https://bugzilla.yoctoproject.org/show_bug.cgi?id=8544)
#
# This affects only a handful of files (fsck.ext4/ext3/... and etc/terminfo/v/vt220)
# so instead of trying to be smart here and create symlinks, just create
# separate copies of all affected files.
ROOTFS_POSTPROCESS_COMMAND_append = " avoid_hardlinks; "
avoid_hardlinks () {
    find ${IMAGE_ROOTFS} -type f -links +1 | while read linked; do
        cp -a $linked $linked.tmp
        mv $linked.tmp $linked
    done
}

Unfortunately this is still incomplete. The image that I am testing with has extended attributes, and fsck.ext4 complains about one of those ("Extended attribute in inode 20327 has a hash (468129694) which is invalid"). The "Failed to optimize directory ??? (17): Corrupt extent header." also still occurs. Furthermore, there are two problems with "Free blocks count wrong".
Comment 2 Patrick Ohly 2015-10-20 06:03:06 UTC
This problem occurs regularly when booting on MinnowMax with an image using systemd, because in that case the /etc/timestamp mechanism for updating the system time early in the boot process is missing and therefore fsck.ext4 always runs a full check because the time stamp on the superblock is in the future.
Comment 3 Patrick Ohly 2015-10-20 08:22:54 UTC
One additional discovery: it only happens when these hardlinks also have xattrs, security.ima in our case.

The layer adding these xattrs is meta-security-smack from https://github.com/01org/meta-intel-iot-security

That means it cannot be reproduced with OE-core or Poky out-of-the-box. If someone wants to investigate further, I can try to reproduce it with Poky + these layers and provide instructions.
Comment 4 Patrick Ohly 2015-10-20 08:30:49 UTC
Hmm, double checking with an image using xattr+hardlinks via "fsck.ext4 -f" called directly from init=/bin/sh did not show the corruption.

Seems that I need to investigate further where exactly the FS gets corrupted.
Comment 5 Patrick Ohly 2015-10-20 09:38:00 UTC
Now I know how reproduce the issue:
1. system time must be incorrect
2. fsck.ext4 must be allowed to fix the superblock

That leads to the "Extended attribute in inode" error (and the other errors that I mentioned before) directly in the fsck.ext4 run where it is updating the time stamp of the superblock.

So it could be fsck.ext4 itself which corrupts the image. Not sure whether it depends on creating the image with mkfs.ext4.

I checked again whether it depends on xattrs. It still occurs when I have less of them (i.e. no ima.security, in particular not on the hard links) but as there are some xattrs elsewhere (Smack labels on /etc, for example) this still might make a difference.
Comment 6 Patrick Ohly 2015-10-20 11:09:31 UTC
I can confirm that fsck.ext4 from current e2fsprogs master (= 956b0f18a5) no longer causes the problems that I ran into.

I suggest updating OE-core master to that version as soon as possible, followed by an update to 1.43 once it is released.
Comment 7 Robert Yang 2015-10-20 11:15:30 UTC
(In reply to comment #6)
> I can confirm that fsck.ext4 from current e2fsprogs master (= 956b0f18a5) no
> longer causes the problems that I ran into.
> 
> I suggest updating OE-core master to that version as soon as possible,
> followed by an update to 1.43 once it is released.

I think we can upgrade it in YP 2.1, for YP 2.0, there isn't anything wrong AFAIK.
Comment 8 Paul Eggleton 2015-10-20 14:58:23 UTC
Surely there must be something wrong or we'd not be having this discussion.

Is there a specific patch in e2fsprogs master that fixes the issue which we can backport?
Comment 9 Robert Yang 2015-10-20 15:38:21 UTC
(In reply to comment #8)
> Surely there must be something wrong or we'd not be having this discussion.
> 
> Is there a specific patch in e2fsprogs master that fixes the issue which we
> can backport?

Patrick doesn't use oe-core's version of e2fsprogs, though oe-core's doesn't support xattrs when mke2fs -d. So I think that there isn't wrong for YP 2.0, I will upgrade e2fsprogs to git version in YP 2.1 so that it supports xattrs, and I will move critical to major.
Comment 10 Patrick Ohly 2015-10-20 20:10:08 UTC
I can reproduce some of the problems also with Poky 2.0 = jethro revision 8578bc17b54da97d42b257b3c8f398ac52b91a9f.

I built core-image-minimal for qemux86 with the following changes in local.conf:
CORE_IMAGE_EXTRA_INSTALL += "e2fsprogs"

Then I ran with:
runqemu qemux86 core-image-minimal ext4 serial nographic kvm 'bootparams=ro init=/bin/sh' 'qemuparams=-monitor none'

In the shell, I ran:

/ # mount -t proc proc /proc
/ # date 010101012014
Wed Jan  1 01:01:00 UTC 2014
/ # fsck.ext4 -n /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
Superblock last write time (Tue Oct 20 19:56:02 2015,
	now = Wed Jan  1 01:01:05 2014) is in the future.
Fix? no

/dev/vda: clean, 799/2624 files, 8204/10492 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
Superblock last write time (Tue Oct 20 19:56:02 2015,
	now = Wed Jan  1 01:01:17 2014) is in the future.
Fix<y>? yes
/dev/vda: clean, 799/2624 files, 8204/10492 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
/dev/vda: clean, 799/2624 files, 8204/10492 blocks
/ # fsck.ext4 -f /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Setting filetype for entry 'e2fsck' in /sbin (499) to 1.
Setting filetype for entry 'fsck.ext4dev' in /sbin (499) to 1.
Setting filetype for entry 'mkfs.ext3' in /sbin (499) to 1.
Setting filetype for entry 'fsck.ext3' in /sbin (499) to 1.
Setting filetype for entry 'fsck.ext2' in /sbin (499) to 1.
Setting filetype for entry 'mkfs.ext4dev' in /sbin (499) to 1.
Setting filetype for entry 'mkfs.ext2' in /sbin (499) to 1.
Setting filetype for entry 'mkfs.ext4' in /sbin (499) to 1.
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda: ***** REBOOT LINUX *****
/dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks

The same "Setting filetype for entry" messages appear. However, the filesystem does not become completely corrupted. Perhaps it would be different for more complex images.

FWIW, I'm still not fully convinced that running fsck is really tested well enough:
* core-image-minimal does not contain it
* .hddimg mounts the rootfs read/write in the initramfs, which prevents running fsck
* mkefidisk.sh seems to do the same (not tested)
Comment 11 Robert Yang 2015-10-21 01:32:01 UTC
(In reply to comment #10)
> I can reproduce some of the problems also with Poky 2.0 = jethro revision
> 8578bc17b54da97d42b257b3c8f398ac52b91a9f.
> 
> I built core-image-minimal for qemux86 with the following changes in
> local.conf:
> CORE_IMAGE_EXTRA_INSTALL += "e2fsprogs"
> 
> Then I ran with:
> runqemu qemux86 core-image-minimal ext4 serial nographic kvm 'bootparams=ro
> init=/bin/sh' 'qemuparams=-monitor none'
> 
> In the shell, I ran:
> 
> / # mount -t proc proc /proc
> / # date 010101012014
> Wed Jan  1 01:01:00 UTC 2014
> / # fsck.ext4 -n /dev/vda

Is /dev/vda mounted ? I will try later.

// Robert

> e2fsck 1.42.9 (28-Dec-2013)
> Superblock last write time (Tue Oct 20 19:56:02 2015,
> 	now = Wed Jan  1 01:01:05 2014) is in the future.
> Fix? no
> 
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Superblock last write time (Tue Oct 20 19:56:02 2015,
> 	now = Wed Jan  1 01:01:17 2014) is in the future.
> Fix<y>? yes
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 -f /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Setting filetype for entry 'e2fsck' in /sbin (499) to 1.
> Setting filetype for entry 'fsck.ext4dev' in /sbin (499) to 1.
> Setting filetype for entry 'mkfs.ext3' in /sbin (499) to 1.
> Setting filetype for entry 'fsck.ext3' in /sbin (499) to 1.
> Setting filetype for entry 'fsck.ext2' in /sbin (499) to 1.
> Setting filetype for entry 'mkfs.ext4dev' in /sbin (499) to 1.
> Setting filetype for entry 'mkfs.ext2' in /sbin (499) to 1.
> Setting filetype for entry 'mkfs.ext4' in /sbin (499) to 1.
> Pass 3: Checking directory connectivity
> Pass 3A: Optimizing directories
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> 
> /dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
> /dev/vda: ***** REBOOT LINUX *****
> /dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks
> 
> The same "Setting filetype for entry" messages appear. However, the
> filesystem does not become completely corrupted. Perhaps it would be
> different for more complex images.
> 
> FWIW, I'm still not fully convinced that running fsck is really tested well
> enough:
> * core-image-minimal does not contain it
> * .hddimg mounts the rootfs read/write in the initramfs, which prevents
> running fsck
> * mkefidisk.sh seems to do the same (not tested)
Comment 12 Patrick Ohly 2015-10-21 07:48:18 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > Then I ran with:
> > runqemu qemux86 core-image-minimal ext4 serial nographic kvm 'bootparams=ro
> > init=/bin/sh' 'qemuparams=-monitor none'
> > 
> > In the shell, I ran:
> > 
> > / # mount -t proc proc /proc
> > / # date 010101012014
> > Wed Jan  1 01:01:00 UTC 2014
> > / # fsck.ext4 -n /dev/vda
> 
> Is /dev/vda mounted ?

Yes, as / That's where /bin/sh and fsck.ext4 come from. But because of the "ro" boot parameter, it was mounted read-only, which enables running fsck on it. 

It is /dev/vda because of the "kvm" runqemu parameter. Otherwise it should be /dev/hda.
Comment 13 Robert Yang 2015-10-23 11:24:27 UTC
Hi Patrick,

Would you please help to test the patch ? It is based on YP 2.0's master branch:

  git://git.openembedded.org/openembedded-core-contrib rbt/e2fsprogs
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/e2fsprogs

Robert Yang (1):
  e2fsprogs: backport a patch to fix filetype for hardlink
Comment 14 Robert Yang 2015-10-23 11:25:50 UTC
Now you may get:

e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/hda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/hda: ***** REBOOT LINUX *****

Note the "Pass 3A: Optimizing directories", it is fine, from "man e2fsck":

    e2fsck may sometimes optimize a few directories --- for example, if
    directory indexing is enabled and a directory is not indexed and would
    benefit from being indexed, or if the index structures are corrupted
    and need to be rebuilt.
Comment 15 Patrick Ohly 2015-10-23 16:11:41 UTC
Do you want me to validate this in OE-core or with xattrs and thus the e2fsprogs from https://github.com/01org/meta-intel-iot-security?

I trust that you made no mistake when testing and that the patch really addresses the problem in OE-core.

More interesting would be whether it also fixes the more severe corruption seen when building images with xattrs. I tried testing that, but the patch does not apply when using the (patched?!) e2fsprogs from the security layer.

The original upstream patch (a433db04d054a4c949c816b28cfa9952b35e0b0a) applies. It seems to address all issues that I was seeing; at least I cannot reproduce them anymore.
Comment 16 Patrick Ohly 2015-10-23 18:42:08 UTC
I spoke to soon. The commit fixes the hardlink problem (obviously), but when fsck also changes the time stamp of the superblock, the fs still gets corrupted in our image. I hadn't reset the time in my previous test.
Comment 17 Robert Yang 2015-10-24 06:44:53 UTC
(In reply to comment #15)
> Do you want me to validate this in OE-core or with xattrs and thus the
> e2fsprogs from https://github.com/01org/meta-intel-iot-security?
> 
> I trust that you made no mistake when testing and that the patch really
> addresses the problem in OE-core.
> 
> More interesting would be whether it also fixes the more severe corruption
> seen when building images with xattrs. I tried testing that, but the patch
> does not apply when using the (patched?!) e2fsprogs from the security layer.
> 
> The original upstream patch (a433db04d054a4c949c816b28cfa9952b35e0b0a)

That's the commit that I backported to YP 2.0.

> applies. It seems to address all issues that I was seeing; at least I cannot
> reproduce them anymore.
Comment 18 Robert Yang 2015-10-24 08:00:55 UTC
(In reply to comment #16)
> I spoke to soon. The commit fixes the hardlink problem (obviously), but when
> fsck also changes the time stamp of the superblock, the fs still gets
> corrupted in our image. I hadn't reset the time in my previous test.

Hi,

I tried the layer meta-intel-iot-security/meta-security-smack after the patch applied to e2fsprogs_git.bb:

OVERRIDES .= ":smack"
DISTRO_FEATURES_append = " smack"

# Enable systemd.
DISTRO_FEATURES_append = " pam"
DISTRO_FEATURES_append += " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""
# CORE_IMAGE_EXTRA_INSTALL += "systemd-analyze"

# Need Smack support in file utilities.
CORE_IMAGE_EXTRA_INSTALL += "coreutils e2fsprogs"

# Having Smack utilities is useful.
CORE_IMAGE_EXTRA_INSTALL += "smack-userspace"

There isn't anything run, or did I miss anything ?
$ bitbake core-image-minimal

/ # mount -t proc proc /proc
/ # date 010101012014
Wed Jan  1 01:01:00 UTC 2014
/ # fsck.ext4 -n /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last write time (Sat Oct 24 07:57:39 2015,
        now = Wed Jan  1 01:01:08 2014) is in the future.
Fix? no

/dev/vda: clean, 1920/17352 files, 54296/69276 blocks
/ # fsck.ext4 -y /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last write time (Sat Oct 24 07:57:39 2015,
        now = Wed Jan  1 01:01:20 2014) is in the future.
Fix? yes

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda: ***** REBOOT LINUX *****
/dev/vda: 1920/17352 files (0.8% non-contiguous), 54308/69276 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.43-WIP (18-May-2014)
/dev/vda: clean, 1920/17352 files, 54308/69276 blocks
/ # fsck.ext4 -f /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vda: 1920/17352 files (0.8% non-contiguous), 54308/69276 blocks


(In reply to comment #16)
> I spoke to soon. The commit fixes the hardlink problem (obviously), but when
> fsck also changes the time stamp of the superblock, the fs still gets
> corrupted in our image. I hadn't reset the time in my previous test.
Comment 19 Patrick Ohly 2015-10-26 09:36:06 UTC
With Smack enabled, you should see some xattrs, for example on /etc. Check with "bitbake -c devshell core-image-minimal" and "getfattr -d -m . /etc" (getfattr from the "attr" package on Debian-based distros).

I don't remember whether I had fatal errors in that case alone. My latest test was with IMA image signing enabled, which sets xattrs on all files. For that, follow the instructions in meta-integrity/README. It should be enough to add to local.conf:
  INHERIT += "ima-evm-rootfs"
Comment 20 Robert Yang 2015-10-27 02:48:08 UTC
(In reply to comment #19)
> With Smack enabled, you should see some xattrs, for example on /etc. Check
> with "bitbake -c devshell core-image-minimal" and "getfattr -d -m . /etc"
> (getfattr from the "attr" package on Debian-based distros).
> 
> I don't remember whether I had fatal errors in that case alone. My latest
> test was with IMA image signing enabled, which sets xattrs on all files. For
> that, follow the instructions in meta-integrity/README. It should be enough
> to add to local.conf:
>   INHERIT += "ima-evm-rootfs"

It works well for me, too:

# getfattr -d -m . /etc
getfattr: Removing leading '/' from absolute path names
# file: etc
security.SMACK64="System::Shared"
security.SMACK64TRANSMUTE="TRUE"

/ # mount -t proc proc /proc
/ # date 010101012014
Wed Jan  1 01:01:00 UTC 2014
/ # fsck.ext4 -n /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last write time (Tue Oct 27 02:41:48 2015,
        now = Wed Jan  1 01:01:03 2014) is in the future.
Fix? no

/dev/vda: clean, 1926/17352 files, 55718/69348 blocks
/ # fsck.ext4 -y /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last write time (Tue Oct 27 02:41:48 2015,
        now = Wed Jan  1 01:01:11 2014) is in the future.
Fix? yes

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda: ***** REBOOT LINUX *****
/dev/vda: 1926/17352 files (0.7% non-contiguous), 55729/69348 blocks
/ # fsck.ext4 -f /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vda: 1926/17352 files (0.7% non-contiguous), 55729/69348 blocks
Comment 22 Alexandru Georgescu 2015-10-28 12:47:03 UTC
(In reply to comment #21)
> Merged into YP 2.0:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> ?h=jethro&id=c1d5e897909bc659eee6151d097781dd3c7931c7

Hi Robert,

I am trying to verify the fix, but I can't tell if it's fixed or not. Maybe you can tell me how to try and reproduce the issue.
I am using jethro/8a0d8eee432924433c3e70198aaeab3161476c5f


I built core-image-minimal for qemux86 with the following changes in local.conf:
CORE_IMAGE_EXTRA_INSTALL += "e2fsprogs"

Then I ran with:
runqemu qemux86 core-image-minimal ext4 serial nographic kvm 'bootparams=ro init=/bin/sh' 'qemuparams=-monitor none'

In the shell, I ran:


/ # date
Wed Oct 28 12:11:58 UTC 2015
/ # mount -t proc proc /proc
/ # date 010101012014
Wed Jan  1 01:01:00 UTC 2014
/ # fsck.ext
fsck.ext2     fsck.ext3     fsck.ext4     fsck.ext4dev
/ # fsck.ext4 -n /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
Superblock last write time (Wed Oct 28 12:11:21 2015,
        now = Wed Jan  1 01:01:10 2014) is in the future.
Fix? no

/dev/vda: clean, 799/2624 files, 8204/10492 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
Superblock last write time (Wed Oct 28 12:11:21 2015,
        now = Wed Jan  1 01:01:28 2014) is in the future.
Fix<y>? yes
/dev/vda: clean, 799/2624 files, 8204/10492 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
/dev/vda: clean, 799/2624 files, 8204/10492 blocks
/ # fsck.ext4 -f /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda: ***** REBOOT LINUX *****
/dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks
/ # mount -
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 8.9M      6.7M      1.5M  81% /
devtmpfs                119.6M         0    119.6M   0% /dev
/ #


Thanks,
AlexG
Comment 23 Alexandru Georgescu 2015-10-28 12:59:54 UTC
(In reply to comment #22)
> (In reply to comment #21)
> > Merged into YP 2.0:
> > 
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> > ?h=jethro&id=c1d5e897909bc659eee6151d097781dd3c7931c7
> 
> Hi Robert,
> 
> I am trying to verify the fix, but I can't tell if it's fixed or not. Maybe
> you can tell me how to try and reproduce the issue.
> I am using jethro/8a0d8eee432924433c3e70198aaeab3161476c5f
> 
> 
> I built core-image-minimal for qemux86 with the following changes in
> local.conf:
> CORE_IMAGE_EXTRA_INSTALL += "e2fsprogs"
> 
> Then I ran with:
> runqemu qemux86 core-image-minimal ext4 serial nographic kvm 'bootparams=ro
> init=/bin/sh' 'qemuparams=-monitor none'
> 
> In the shell, I ran:
> 
> 
> / # date
> Wed Oct 28 12:11:58 UTC 2015
> / # mount -t proc proc /proc
> / # date 010101012014
> Wed Jan  1 01:01:00 UTC 2014
> / # fsck.ext
> fsck.ext2     fsck.ext3     fsck.ext4     fsck.ext4dev
> / # fsck.ext4 -n /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Superblock last write time (Wed Oct 28 12:11:21 2015,
>         now = Wed Jan  1 01:01:10 2014) is in the future.
> Fix? no
> 
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Superblock last write time (Wed Oct 28 12:11:21 2015,
>         now = Wed Jan  1 01:01:28 2014) is in the future.
> Fix<y>? yes
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 -f /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 3A: Optimizing directories
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> 
> /dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
> /dev/vda: ***** REBOOT LINUX *****
> /dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks
> / # mount -
> / # df -h
> Filesystem                Size      Used Available Use% Mounted on
> /dev/root                 8.9M      6.7M      1.5M  81% /
> devtmpfs                119.6M         0    119.6M   0% /dev
> / #
> 
> 
> Thanks,
> AlexG

reminder for myself to investigate the kernel panic that I encountered when exiting the session:


/dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda: ***** REBOOT LINUX *****
/dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks
/ # mount -
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 8.9M      6.7M      1.5M  81% /
devtmpfs                119.6M         0    119.6M   0% /dev
/ # exit
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000

CPU: 0 PID: 1 Comm: sh Not tainted 4.1.8-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
 00000000 00000000 cf89ff28 c186350e c1b68fa0 cf89ff40 c1861b93 cf13e438
 c1b68fa0 cf13e438 cf8a0000 cf89ff8c c1052994 c1a62f00 00000000 c1052c4b
 cf8a8000 00000001 00000005 cf8a0000 00000001 cf13e438 00000000 00000000
Call Trace:
 [<c186350e>] dump_stack+0x4b/0x75
 [<c1861b93>] panic+0x81/0x17b
 [<c1052994>] do_exit+0xab4/0xac0
 [<c1052c4b>] ? SyS_wait4+0x6b/0xc0
 [<c1052a11>] do_group_exit+0x31/0x90
 [<c1052a86>] SyS_exit_group+0x16/0x20
 [<c1869e0e>] syscall_call+0x7/0x7
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
Comment 24 Bogdan Alexandru Voiculescu 2015-10-28 16:01:54 UTC
Trying to reproduce with smack layer:



First attempt:
==================

Write protecting the kernel text: 8680k
Write protecting the kernel read-only data: 2960k
/bin/sh: can't access tty; job control turned off
/ # date
Thu Oct 29 01:46:51 UTC 2015
/ # mount -t proc proc /proc
/ # date 010101012014
Wed Jan  1 01:01:00 UTC 2014
/ # fsck.ext
/bin/sh: fsck.ext: not found
/ # fs
fsck                   fsck.ext4dev           fsfreeze
fsck.ext2              fsck.minix             fstrim
fsck.ext3              fsck.minix.util-linux
fsck.ext4              fsck.util-linux
/ # fsck.ext4 -n /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last mount time (Thu Oct 29 01:43:03 2015,
	now = Wed Jan  1 01:01:14 2014) is in the future.
Fix? no

Superblock last write time (Thu Oct 29 01:43:03 2015,
	now = Wed Jan  1 01:01:14 2014) is in the future.
Fix? no

/dev/vda: clean, 2074/18504 files, 59227/73729 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last mount time (Thu Oct 29 01:43:03 2015,
	now = Wed Jan  1 01:01:21 2014) is in the future.
Fix<y>? yes
Superblock last write time (Thu Oct 29 01:43:03 2015,
	now = Wed Jan  1 01:01:21 2014) is in the future.
Fix<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Setting filetype for entry 'mkfs.ext3' in /sbin (471) to 1.
Setting filetype for entry 'mkfs.ext4dev' in /sbin (471) to 1.
Setting filetype for entry 'e2fsck' in /sbin (471) to 1.
Setting filetype for entry 'vt200' in /etc/terminfo/v (1968) to 1.
Setting filetype for entry 'fsck.ext4' in /sbin (471) to 1.
Setting filetype for entry 'fsck.ext4dev' in /sbin (471) to 1.
Setting filetype for entry 'mke2fs' in /sbin (471) to 1.
Setting filetype for entry 'mkfs.ext4' in /sbin (471) to 1.
Setting filetype for entry 'fsck.ext3' in /sbin (471) to 1.
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda: ***** REBOOT LINUX *****
/dev/vda: 2074/18504 files (0.8% non-contiguous), 59239/73729 blocks
/ # fsck.ext4 -f /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vda: 2074/18504 files (0.8% non-contiguous), 59239/73729 blocks
/ # mount -
mount: can't find - in /etc/fstab
/ # mount 
/dev/vda on / type ext4 (ro,relatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=122404k,nr_inodes=30601,mode=755)
proc on /proc type proc (rw,relatime)
/ # df -h 
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        66M   52M  9.2M  85% /
devtmpfs        120M     0  120M   0% /dev


==================

Then I restarted the qemu and repeated the steps: 
==================

EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) readonly on device 253:0.
devtmpfs: mounted
Freeing unused kernel memory: 732K (c1c0c000 - c1cc3000)
Write protecting the kernel text: 8680k
Write protecting the kernel read-only data: 2960k
/bin/sh: can't access tty; job control turned off
/ # date
Thu Oct 29 01:59:50 UTC 2015
/ # mount -t proc proc /proc
/ # date 010101012014
Wed Jan  1 01:01:00 UTC 2014
/ # fs
fsck                   fsck.ext4dev           fsfreeze
fsck.ext2              fsck.minix             fstrim
fsck.ext3              fsck.minix.util-linux
fsck.ext4              fsck.util-linux
/ # fsck
fsck from util-linux 2.26.2
/ # fsck.ext4 -n /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last mount time is in the future.
	(by less than a day, probably due to the hardware clock being incorrectly set)  Fix? no

Superblock last write time is in the future.
	(by less than a day, probably due to the hardware clock being incorrectly set).  Fix? no

/dev/vda: clean, 2074/18504 files, 59239/73729 blocks
/ # fsck.ext4 -y /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Superblock last write time is in the future.
	(by less than a day, probably due to the hardware clock being incorrectly set).  Fix? yes

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vda: 2074/18504 files (0.8% non-contiguous), 59239/73729 blocks
/ # fsck.ext4 /dev/vda
e2fsck 1.43-WIP (18-May-2014)
/dev/vda: clean, 2074/18504 files, 59239/73729 blocks
/ # fsck.ext4 -f /dev/vda
e2fsck 1.43-WIP (18-May-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vda: 2074/18504 files (0.8% non-contiguous), 59239/73729 blocks
/ # date
Wed Jan  1 01:01:51 UTC 2014

 
So please advise!
Comment 25 Robert Yang 2015-10-29 02:02:18 UTC
(In reply to comment #23)
> (In reply to comment #22)
> > (In reply to comment #21)
> > > Merged into YP 2.0:
> > > 
> > > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/
> > > ?h=jethro&id=c1d5e897909bc659eee6151d097781dd3c7931c7
> > 
> > Hi Robert,
> > 
> > I am trying to verify the fix, but I can't tell if it's fixed or not. Maybe
> > you can tell me how to try and reproduce the issue.
> > I am using jethro/8a0d8eee432924433c3e70198aaeab3161476c5f
> > 
> > 
> > I built core-image-minimal for qemux86 with the following changes in
> > local.conf:
> > CORE_IMAGE_EXTRA_INSTALL += "e2fsprogs"
> > 
> > Then I ran with:
> > runqemu qemux86 core-image-minimal ext4 serial nographic kvm 'bootparams=ro
> > init=/bin/sh' 'qemuparams=-monitor none'
> > 
> > In the shell, I ran:
> > 
> > 
> > / # date
> > Wed Oct 28 12:11:58 UTC 2015
> > / # mount -t proc proc /proc
> > / # date 010101012014
> > Wed Jan  1 01:01:00 UTC 2014
> > / # fsck.ext
> > fsck.ext2     fsck.ext3     fsck.ext4     fsck.ext4dev
> > / # fsck.ext4 -n /dev/vda
> > e2fsck 1.42.9 (28-Dec-2013)
> > Superblock last write time (Wed Oct 28 12:11:21 2015,
> >         now = Wed Jan  1 01:01:10 2014) is in the future.
> > Fix? no
> > 
> > /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> > / # fsck.ext4 /dev/vda
> > e2fsck 1.42.9 (28-Dec-2013)
> > Superblock last write time (Wed Oct 28 12:11:21 2015,
> >         now = Wed Jan  1 01:01:28 2014) is in the future.
> > Fix<y>? yes
> > /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> > / # fsck.ext4 /dev/vda
> > e2fsck 1.42.9 (28-Dec-2013)
> > /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> > / # fsck.ext4 -f /dev/vda
> > e2fsck 1.42.9 (28-Dec-2013)
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Pass 3: Checking directory connectivity
> > Pass 3A: Optimizing directories
> > Pass 4: Checking reference counts
> > Pass 5: Checking group summary information
> > 
> > /dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
> > /dev/vda: ***** REBOOT LINUX *****
> > /dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks
> > / # mount -
> > / # df -h
> > Filesystem                Size      Used Available Use% Mounted on
> > /dev/root                 8.9M      6.7M      1.5M  81% /
> > devtmpfs                119.6M         0    119.6M   0% /dev
> > / #
> > 
> > 
> > Thanks,
> > AlexG
> 
> reminder for myself to investigate the kernel panic that I encountered when
> exiting the session:
> 
> 
> /dev/vda: ***** FILE SYSTEM WAS MODIFIED *****
> /dev/vda: ***** REBOOT LINUX *****
> /dev/vda: 799/2624 files (0.8% non-contiguous), 8206/10492 blocks
> / # mount -
> / # df -h
> Filesystem                Size      Used Available Use% Mounted on
> /dev/root                 8.9M      6.7M      1.5M  81% /
> devtmpfs                119.6M         0    119.6M   0% /dev
> / # exit
> Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
> 
> CPU: 0 PID: 1 Comm: sh Not tainted 4.1.8-yocto-standard #1
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
>  00000000 00000000 cf89ff28 c186350e c1b68fa0 cf89ff40 c1861b93 cf13e438
>  c1b68fa0 cf13e438 cf8a0000 cf89ff8c c1052994 c1a62f00 00000000 c1052c4b
>  cf8a8000 00000001 00000005 cf8a0000 00000001 cf13e438 00000000 00000000
> Call Trace:
>  [<c186350e>] dump_stack+0x4b/0x75
>  [<c1861b93>] panic+0x81/0x17b
>  [<c1052994>] do_exit+0xab4/0xac0
>  [<c1052c4b>] ? SyS_wait4+0x6b/0xc0
>  [<c1052a11>] do_group_exit+0x31/0x90
>  [<c1052a86>] SyS_exit_group+0x16/0x20
>  [<c1869e0e>] syscall_call+0x7/0x7
> Kernel Offset: disabled
> ---[ end Kernel panic - not syncing: Attempted to kill init!
> exitcode=0x00000000

The kernel panic issue doesn't related to fsck, you can get the issue everytime when you you exit it (w/o fsck). I think that it is because of 'bootparams=ro init=/bin/sh', if you remove it from command line, it will be OK.

Use jethro to test it is OK, if you want to use meta-security-smack, you need apply another similar patch to its e2fsprogs recipe, the layer has its own e2fsprogs.
Comment 26 Patrick Ohly 2015-10-29 07:47:50 UTC
The kernel panic is caused by the init process exiting. That's normal in this situation, because without that process, what it is kernel supposed to run? Just power off instead.

We've also seen the same panic when the init process cannot be loaded because of severe file system corruption. That also is just a follow-up error can then can be ignored.
Comment 27 Patrick Ohly 2015-10-29 12:21:53 UTC
(In reply to comment #22)
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Superblock last write time (Wed Oct 28 12:11:21 2015,
>         now = Wed Jan  1 01:01:28 2014) is in the future.
> Fix<y>? yes
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> /dev/vda: clean, 799/2624 files, 8204/10492 blocks
> / # fsck.ext4 -f /dev/vda
> e2fsck 1.42.9 (28-Dec-2013)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 3A: Optimizing directories
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information

That shows that the hardlink issue is fixed.

(In reply to comment #24)
> Trying to reproduce with smack layer:
...
> /dev/vda: clean, 2074/18504 files, 59227/73729 blocks
> / # fsck.ext4 /dev/vda
> e2fsck 1.43-WIP (18-May-2014)
> Superblock last mount time (Thu Oct 29 01:43:03 2015,
> 	now = Wed Jan  1 01:01:21 2014) is in the future.
> Fix<y>? yes
> Superblock last write time (Thu Oct 29 01:43:03 2015,
> 	now = Wed Jan  1 01:01:21 2014) is in the future.
> Fix<y>? yes
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Setting filetype for entry 'mkfs.ext3' in /sbin (471) to 1.
> Setting filetype for entry 'mkfs.ext4dev' in /sbin (471) to 1.
> Setting filetype for entry 'e2fsck' in /sbin (471) to 1.
> Setting filetype for entry 'vt200' in /etc/terminfo/v (1968) to 1.
> Setting filetype for entry 'fsck.ext4' in /sbin (471) to 1.
> Setting filetype for entry 'fsck.ext4dev' in /sbin (471) to 1.
> Setting filetype for entry 'mke2fs' in /sbin (471) to 1.
> Setting filetype for entry 'mkfs.ext4' in /sbin (471) to 1.
> Setting filetype for entry 'fsck.ext3' in /sbin (471) to 1.
> Pass 3: Checking directory connectivity
> Pass 3A: Optimizing directories
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information

This is what you get when e2fsprogs has the hardlink bug.

It's expected that you get this when using the current meta-intel-iot-security/meta-smack layer, because it has the broken e2fsprogs.
Comment 28 Bogdan Alexandru Voiculescu 2015-10-29 12:42:44 UTC
Thank you, very much, Patrick! 
Do we need to wait for the patch for e2e2fsprogs from smack layer, or we can close the bug, since smack it is outside our project?
Comment 29 Patrick Ohly 2015-10-29 13:09:44 UTC
Feel free to close this bug.
Comment 30 Bogdan Alexandru Voiculescu 2015-10-29 15:29:01 UTC
Thank you, Patrick.