openembedded-core/meta/classes-recipe/image_types.bbclass: oe_mkext234fs () { [Image creation] # Error codes 0-3 indicate successfull operation of fsck (no errors or errors corrected) fsck.$fstype -pvfD ${IMGDEPLOYDIR}/${IMAGE_NAME}.$fstype || [ $? -le 3 ] ] This was added in order to prevent an extra reboot on the target: https://git.openembedded.org/openembedded-core/commit/?id=a93d0059341 However, this reduces reproducibility due to an increased image delta: https://youtu.be/f_mwJH6mEHA?t=685 We have verified that on our image, delta is decreased after removing this check: Patch Size Matrix Before Check Removal (values in KB) mender1 mender2 mender-verity1 mender-verity2 mender1 0 6840 6840 6840 mender2 6840 0 6840 6840 mender-verity1 6840 6840 0 6840 mender-verity2 6840 6840 6840 0 Patch Size Matrix After Check Removal (values in KB): mender1 mender2mender-verity1mender-verity2 mender1 0 6824 6824 6824 mender2 6824 0 6824 6824 mender-verity1 6824 6824 0 6824 mender-verity2 6824 6824 6824 0 For us, reproducibility is much more desirable than preventing an extra reboot on our targets. It may be prudent to make this check optional by adding the option to configure whether or not it's run. I would think others might be in a similar position.
I replied to the patch on the list, copy-pasting here: There’s slightly more information in the bug report than here, which links to a video from YPS 2024.12 talking about how fsck will modify timestamps in file systems, so whilst the initial ext4 from mkfs is reproducible, we do a fsck to clear flags and they’re no longer bit-identical. Is this the only source of non-determinism that you’re observing? I don’t think adding an option is the right thing here as you’re swapping one problem (non-reproducible ext4) with another (filesystem dirty, needs a reboot). As the video shows, we should be passing timestamp information at construction time to avoid this problem. Would you be able to work on a patch to do that instead?
(In reply to Ross Burton from comment #1) > I replied to the patch on the list, copy-pasting here: > > There’s slightly more information in the bug report than here, which links > to a video from YPS 2024.12 talking about how fsck will modify timestamps in > file systems, so whilst the initial ext4 from mkfs is reproducible, we do a > fsck to clear flags and they’re no longer bit-identical. Is this the only > source of non-determinism that you’re observing? > > I don’t think adding an option is the right thing here as you’re swapping > one problem (non-reproducible ext4) with another (filesystem dirty, needs a > reboot). As the video shows, we should be passing timestamp information at > construction time to avoid this problem. Would you be able to work on a > patch to do that instead? Here is a matrix of image deltas after having eliminated, among other things, delta caused by differing timestamps: Patch Size Matrix (values in KB): with-fsck1 with-fsck2 with-fsck3without-fsck1without-fsck2without-fsck3 with-fsck1 0 196 192 192 192 192 with-fsck2 196 0 192 192 192 192 with-fsck3 196 192 0 192 192 192 without-fsck1 196 196 196 0 172 172 without-fsck2 196 196 196 172 0 172 without-fsck3 196 196 196 172 172 0 The only difference between these builds was the presence of the aforementioned fsck line. To my knowledge, nobody thus far has been able to describe why this command increases delta in such a way. The linked video from YPS 2024.12 recommends removal of the fsck command at this timestamp: https://youtu.be/f_mwJH6mEHA?si=iX0E1XI7I5HfCBeJ&t=881.
The usage of fsck.$fstype -pvfD at the end of oe_mkext234fs causes non-determinism in two ways: 1. Superblock timestamps — e2fsck stamps s_lastcheck and s_wtime with the current time on every run. E2FSPROGS_FAKE_TIME doesn't help here: it's honored for mke2fs's s_mkfs_time but not for e2fsck's last-check/write times. 2. -D directory optimization — it re-sorts/reindexes every directory using the filesystem's s_hash_seed, which mke2fs assigns randomly on this path. So each build gets different htree blocks, and since -D touches every directory it spreads that randomness widely. Compounding it at the mke2fs level: the random s_hash_seed (which -D depends on) and the random filesystem UUID — and with metadata_csum and no metadata_csum_seed, the UUID seeds every metadata checksum, so one differing UUID re-seeds all of them and looks like a large delta even when the layout is identical. Notably, wic's ext4 path already handles all of this (scripts/lib/wic/partition.py:get_hash_seed_ext4 sets E2FSPROGS_FAKE_TIME + a deterministic -E hash_seed, then normalizes the superblock times with debugfs) — the direct oe_mkext234fs path just never got the same treatment. Mirroring it there (fake-time + deterministic hash_seed + a post-fsck debugfs time fixup, UUID left to the user via EXTRA_IMAGECMD as in wic) makes repeated builds byte-identical.
Patch got merged: https://git.openembedded.org/openembedded-core/commit/?id=8d1d548e36fa8426f20f43a514f0dd60b699f86b Levi, can we close this bug?
I think so, closing. Levi, please reopen if needed.