Bug 13994 - wic is modifying /etc/fstab in place, which races with other image type tasks
Summary: wic is modifying /etc/fstab in place, which races with other image type tasks
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.3 M2
Assignee: Paul Barker
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-08-03 05:26 UTC by Alexander Kanavin
Modified: 2021-02-11 16:21 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kanavin 2020-08-03 05:26:53 UTC
Since

commit 58589bc5382b5c77825bc963a1c84a4e8f54498f
Author: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Date:   Thu Jul 18 15:13:56 2019 +0200

    wic: Fix (again) partition files UIDs on multi rootfs images


rootfs/etc/fstab is modified in place during do_image_wic. If other image types are enabled (such as ext4), the respective do_image_* tasks create image files at the same time as this modification is happening, which leads to unpredictable outcome (fstab may or may not be modified by wic in the output).

I do not have a quick solution (inserting task dependencies does not sound appealing; making a rootfs copy is slow and is causing pseudo issues that the commit above is aiming to solve), but wanted to have a record for the problem neverthless. 

One possible option is to mount the wic-made rootfs file after it has been created, then modify fstab there and only there.
Comment 1 Alexander Kanavin 2020-08-03 05:27:24 UTC
Ricardo, FYI.
Comment 2 Ricardo Ribalda 2020-08-03 07:36:12 UTC
I believe that after the latest changes

https://git.openembedded.org/openembedded-core/commit/?id=fd739c15cdba221d9d497d3402b7d929c0be2ca4 (among others)

Pseudo should be playing nice with wic, but as you mentioned, that involves copying the partition and that is slow.

Mounting the wic rootfs after its creations seems like the best cadidate, although I do not know if all the filesystem will allow you to do that.
Comment 3 Jan-Simon Moeller 2020-10-06 09:27:35 UTC
I was just hitting this as well here. Spuriously b/c of the parallelism of do_image_ext4 + do_image_tar + do_image_wic  running at the same time.

I think the mounting is the better way, but if that is not feasible, why not make do_image_wic run *last* e.g. after do_image_ext4 and do_image_tar ?
Comment 4 Jan-Simon Moeller 2020-10-06 13:27:02 UTC
e.g. add to image_types_wic.bbclass:


# make sure that ext4 and tar are completely written out before we modify files to create the wic format (e.g. /etc/fstab !)

do_image_wic[deptask] += "${@bb.utils.contains('IMAGE_FSTYPES', 'ext4', '%s:do_image_ext4' % d.getVar('PN'), '', d)}"

do_image_wic[deptask] += "${@bb.utils.contains('IMAGE_FSTYPES', 'tar', '%s:do_image_tar' % d.getVar('PN'), '', d)}"


Not sure what other image-fstype would need it. 
Still mounting before modification is indeed better.
Comment 5 Alexander Kanavin 2020-10-06 13:46:27 UTC
Hardcoding task dependencies like this is too brittle, as there is no mechanism to ensure all possible image types are accounted for, and there's quite a bit more than ext4 and tar (see classes/image_types.bbclass).
Comment 6 Jan-Simon Moeller 2020-10-06 22:44:38 UTC
Agree, hardcoding is not the way to go.
Comment 7 Paul Barker 2020-12-21 14:42:09 UTC
I can see 3 possible approaches here, all imperfect:

1) We can easily modify fstab inside the filesystem image after it is created using mcopy (for vfat) or debugfs (for ext2/3/4). But I can't see an equivalent tool for btrfs.

2) We can copy the rootfs into a temporary directory so that fstab can be modified without interfering with other do_image_* tasks. This must be done as a full copy though along with copying the pseudo db, we can't use copyhardlinktree for the same reason explained in the comments added in https://git.openembedded.org/openembedded-core/commit/?id=e8c21c6ebaebde88151697381bdb2452f1171090. So this option would use more space and slow down the build.

3) The final alternative is to ensure that no other do_image_* tasks can run in parallel with do_image_wic. Rather than hardcoding dependencies we could use lockfiles for this though it would have the unfortunate side effect of making all do_image_* functions run in series even if no fstab modification occurs. We don't have support in bitbake for a more complex read/write lock arrangement.

I think my preference would be to check the fstype and use approach (1) for vfat & ext2/3/4, falling back to approach (2) for btrfs.

Any thoughts Ricardo or Alex?
Comment 8 Tom Rini 2020-12-21 15:39:34 UTC
I think approach 3, if documented / sufficiently clear would also help expose a more generic way to fix other race conditions that can occur (for example when using dm-verity and also needing to deal with that fitImage+initramfs being in an image).
Comment 9 Ricardo Ribalda 2020-12-21 17:21:06 UTC
(In reply to comment #8)
> I think approach 3, if documented / sufficiently clear would also help
> expose a more generic way to fix other race conditions that can occur (for
> example when using dm-verity and also needing to deal with that
> fitImage+initramfs being in an image).

I am with Tony, approach 1) will require a fix for every single rfs type and 2) is a tremendous penalty to the overall performance

Also, a disclaimer here, I just changed jobs so while I get up to the speed with the new role I would not have much time to work on this sorry :(
Comment 10 Paul Barker 2020-12-21 17:26:47 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > I think approach 3, if documented / sufficiently clear would also help
> > expose a more generic way to fix other race conditions that can occur (for
> > example when using dm-verity and also needing to deal with that
> > fitImage+initramfs being in an image).
> 
> I am with Tony, approach 1) will require a fix for every single rfs type and
> 2) is a tremendous penalty to the overall performance
> 
> Also, a disclaimer here, I just changed jobs so while I get up to the speed
> with the new role I would not have much time to work on this sorry :(

The issue with approach (3) is that it can also add a huge penalty to rebuilds when very little has changed but images need to be recreated. On a previous client project I saw rebuild time for minimal changes (i.e. just the date stamp in os-release changed) go from 5min to 7min (+40% build time) when do_image_tar and do_image_wic had to run in series.

My plan right now is to implement (2) as the default solution and then optimise this where possible with (1), i.e. for vfat and ext2/3/4.