Bug 10712 - Have wic split the rootfs filesystem and distribute specified subdirectories to different partitions
Summary: Have wic split the rootfs filesystem and distribute specified subdirectories ...
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: Future
Assignee: Kristian Amlie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-11-23 14:18 UTC by Kristian Amlie
Modified: 2017-02-16 12:11 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kristian Amlie 2016-11-23 14:18:07 UTC
Originally discussed in mailing list thread: http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129036.html

The motivation is to be able to generate multiple partitions using content from different parts of the rootfs filesystem.

Current proposal is:
This can be done by extending existing rootfs plugin. It should be able
to do 2 things:

- populate content of one rootfs directory to the partition. We can
  extend syntax of --rootfs-dir parameter to specify optional directory path to use

- exclude rootfs directories when populating partitions. I'd propose to
  introduce --exclude-dirs wks parser option to handle this.

Example of wks file with proposed new options:
part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
Comment 1 Kristian Amlie 2016-11-23 14:19:56 UTC
On the mailing list, Maciej Borzecki pointed out that --exclude-dir may be a bad name since we do still want to the directory to be present. Can I suggest --exclude-content-of instead?
Comment 2 Ed Bartosh 2016-11-23 15:47:39 UTC
It makes thanks for me. Thanks for pointing out.
What about a bit shorter name --exclude-content ?
Comment 3 Kristian Amlie 2016-11-24 07:27:10 UTC
--exclude-content it is!
Comment 4 Ed Bartosh 2016-11-24 15:27:28 UTC
Latest proposal is to use this syntax to exclude content or directories:

--exclude-path data/* - exclude content, but keep the directory
--exclude-path data - exclude directory and content
--exclude-path data/tmp/t* - exclude files and directories that match wildcard
--exclude-path data/db/tmp - exclude subdirectory and its content
Comment 5 Kristian Amlie 2016-11-24 15:53:38 UTC
Looks good to me. I'm still awaiting consensus on the mailing list, but I can start with the heart of the problem and make adjustments later.

I'll get started on this tomorrow.
Comment 6 Kristian Amlie 2016-11-24 16:43:52 UTC
After further discussion on mailing list, latest proposal is to use rsync-style:

--exclude-path data/ - exclude content, but keep the directory
--exclude-path data - exclude directory and content
--exclude-path data/db/tmp/ - exclude subdirectory content only
--exclude-path data/db/tmp - exclude subdirectory and its content

This doesn't necessarily invalidate the earlier glob proposal, but I will not focus on that in the first iteration at least.
Comment 7 Ed Bartosh 2016-11-25 12:51:04 UTC
Reassigning to Kristian as he'll be implementing this.
Comment 8 Kristian Amlie 2016-12-30 08:07:35 UTC
This is likely to be merged soon, but there was talk about having an exclude option for the do_image_ext4 and similar functions as well, in addition to wic. We are going to need both in Mender. Should this ticket cover that or should I make a new one?
Comment 9 Kristian Amlie 2017-01-31 13:08:59 UTC
Setting milestone to "Future" for now. This is ready to be merged, but I don't know when the maintainers will do that. It might need a rebase, but I'll look at that after the sysroot issues have died down a bit, I don't think anyone has any spare cycles ATM.
Comment 10 Ed Bartosh 2017-02-02 17:54:13 UTC
Cristian, can you rebase your changes on top of recent master and send them again for review? I'll try to push maintainers to merge them.
Comment 11 Kristian Amlie 2017-02-03 12:03:32 UTC
Thanks, Ed. I will repost shortly, I just need to retest them with the latest poky.
Comment 12 Kristian Amlie 2017-02-06 16:22:11 UTC
Rebased and retested patchset here: http://lists.openembedded.org/pipermail/openembedded-core/2017-February/132280.html
Comment 13 Ed Bartosh 2017-02-06 16:27:47 UTC
Kristian, thank you! I'll try to make sure your patchset is merged this time.
Comment 14 Ed Bartosh 2017-02-16 11:52:15 UTC
the patchset has been merged to master.
Comment 15 Kristian Amlie 2017-02-16 12:11:30 UTC
Thank you, Ed!