Bug 16110

I replied to the patch on the list, copy-pasting here:

There’s slightly more information in the bug report than here, which links to a video from YPS 2024.12 talking about how fsck will modify timestamps in file systems, so whilst the initial ext4 from mkfs is reproducible, we do a fsck to clear flags and they’re no longer bit-identical.  Is this the only source of non-determinism that you’re observing?

I don’t think adding an option is the right thing here as you’re swapping one problem (non-reproducible ext4) with another (filesystem dirty, needs a reboot).  As the video shows, we should be passing timestamp information at construction time to avoid this problem. Would you be able to work on a patch to do that instead?

 (In reply to Ross Burton from comment #1)
> I replied to the patch on the list, copy-pasting here:
> 
> There’s slightly more information in the bug report than here, which links
> to a video from YPS 2024.12 talking about how fsck will modify timestamps in
> file systems, so whilst the initial ext4 from mkfs is reproducible, we do a
> fsck to clear flags and they’re no longer bit-identical.  Is this the only
> source of non-determinism that you’re observing?
> 
> I don’t think adding an option is the right thing here as you’re swapping
> one problem (non-reproducible ext4) with another (filesystem dirty, needs a
> reboot).  As the video shows, we should be passing timestamp information at
> construction time to avoid this problem. Would you be able to work on a
> patch to do that instead?


Here is a matrix of image deltas after having eliminated, among other things, delta caused by differing timestamps:

Patch Size Matrix (values in KB):
              with-fsck1  with-fsck2  with-fsck3without-fsck1without-fsck2without-fsck3
  with-fsck1           0         196         192         192         192         192
  with-fsck2         196           0         192         192         192         192
  with-fsck3         196         192           0         192         192         192
without-fsck1         196         196         196           0         172         172
without-fsck2         196         196         196         172           0         172
without-fsck3         196         196         196         172         172           0

The only difference between these builds was the presence of the aforementioned fsck line. To my knowledge, nobody thus far has been able to describe why this command increases delta in such a way. 

The linked video from YPS 2024.12 recommends removal of the fsck command at this timestamp: https://youtu.be/f_mwJH6mEHA?si=iX0E1XI7I5HfCBeJ&t=881.

The usage of fsck.$fstype -pvfD at the end of oe_mkext234fs causes non-determinism in two ways:

	1. Superblock timestamps — e2fsck stamps s_lastcheck and s_wtime with the current time on every run. E2FSPROGS_FAKE_TIME doesn't help here:
	it's honored for mke2fs's s_mkfs_time but not for e2fsck's last-check/write times.
	2. -D directory optimization — it re-sorts/reindexes every directory using the filesystem's s_hash_seed, which mke2fs assigns randomly on
	this path. So each build gets different htree blocks, and since -D touches every directory it spreads that randomness widely.

Compounding it at the mke2fs level: the random s_hash_seed (which -D depends on) and the random filesystem UUID — and with metadata_csum
and no metadata_csum_seed, the UUID seeds every metadata checksum, so one differing UUID re-seeds all of them and looks like a large delta
even when the layout is identical.

Notably, wic's ext4 path already handles all of this (scripts/lib/wic/partition.py:get_hash_seed_ext4 sets E2FSPROGS_FAKE_TIME + a
deterministic -E hash_seed, then normalizes the superblock times with debugfs) — the direct oe_mkext234fs path just never got the same
treatment. Mirroring it there (fake-time + deterministic hash_seed + a post-fsck debugfs time fixup, UUID left to the user via
EXTRA_IMAGECMD as in wic) makes repeated builds byte-identical.

Patch got merged: https://git.openembedded.org/openembedded-core/commit/?id=8d1d548e36fa8426f20f43a514f0dd60b699f86b

Levi, can we close this bug?

I think so, closing. Levi, please reopen if needed.