Bug 5488 - wic updates
Summary: wic updates
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: deployment (show other bugs)
Version: 1.6
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 1.7
Assignee: Tom Zanussi
QA Contact: Costin
URL:
Whiteboard:
Depends on:
Blocks: 5519
  Show dependency tree
 
Reported: 2013-11-06 15:46 UTC by Tom Zanussi
Modified: 2014-09-29 05:31 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Zanussi 2013-11-06 15:46:59 UTC
In response to developers wanting to add support for uboot and other image types we need to add some infrastructure support as described in the text below (from an e-mail on the subject).

This is expected work that didn't make it into the initial commits.

there are a few things that I think are going to need some additions to the current support:

- squashfs support - so far we only support ext2/3/4 and btrfs
- bootloader additions - support for 'uboot' and 'barebox', probably a new generic 'part' param ('Burn bootloader' section)
- bootimg creation - specialized boot partition image creation, probably a new --source and generalized 'wic' param ('Create boot partition image' section)

To give a quick overview of the overall image creation flow, basically
how it works is that all the partitions defined in the kickstart file
are created as filesystem images and then those images are assembled
into the final image.  Partitions that have a --source specified are
created by a specific source handler in the code (via prepare(), see
below) and the size of those partitions are the size of the filesystem
image created by the prepare() function for the source.  Other
partitions are created and sized explicitly according to the --size
param).

Once all the partitions are created the PartitionedMount
layout_partitions() function in scripts/lib/mic/utils/partitionedfs.py
uses those sizes to create the disk image and dd all the filesystem
partitions into the final disk image.  There's an --align param that
specifies that a given partition should be aligned on an XXXX kb
boundary.  You should be able to use that to create your 4 and 8 Mb
alignments, for example.

So basically the way it works is that for each partition defined by the
kickstart file, the Wic_PartData.prepare() function for that object is
called.  If the partition specified a source (using the --source param
to the part command), the base prepare() function dispatches the
prepare() function to a 'source'-specific prepare() function.

Currently, we only define two 'sources': 'bootimg' and 'rootfs'.

The 'rootfs' source handler's job is to just take the contents of the
passed-in rootfs (this comes from the wic -r command-line param) and
create a root filesystem image.  The name of the rootfs source handler
is prepare_rootfs().  It in turn dispatches to a fstype-specific prepare
handler, currently those are prepare_rootfs_ext() and
prepare_rootfs_btfs().  For this, we'll need to add a new
prepare_rootfs_squashfs().

For the 'bootfs' source handler, we also have two handlers, and these
are dispatched to by 'boot_type', which is defined by DirectImageCreator
get_boot_type() in scripts/lib/mic/imager/direct.py, which is basically
the --fstype defined for the /boot partition. 

To add your own customized source for the boot, the easiest thing to do
would be to add a new --fstype e.g. --fstype=nitrogen6xboot.  The source
of the artifacts to this comes from the wic -b command-line param, and
that's what will be passed to your handler function, and which you can
use to create your bootfs.  In your case, you'd essentially add your own
handler along the lines of prepare_bootimg_efi() in
Wic_PartData.prepare() e.g.:

if self.source == "bootimg" and boot_type == "nitrogen6xboot":
     self.prepare_bootimg_nitrogen6xboot(cr_workdir,
                oe_builddir, bootimg_dir, kernel_dir, native_sysroot)

And do all the steps in the handler that you do in your '# Create boot
partition image' code.

NOTE: we may want to do this by adding a new --source instead - if so,
that's not any harder, but it does mean we need to generalize the wic
commandline to add a new -s (source param) to match the handler up with.
This would be preferred actually, but for now we can do the above (and
assuming we're not concerned with future compatibility).

So that covers your root and boot partitions, the third partition is
essentially just a fixed-size ext4 partition - you can use the --size
param for that.
TODO: If you need to have the size be a variable, we'll also need to add
some support for that.

Also, note that there currently isn't any way for the partitions that
are based on --source be any size other than what they're created as by
prepare(), so the prepare() step should add whatever size it wants.
TODO: we need to be able to add e.g. +100 to a --source partition.
TODO: we also need to implement --grow so a partition can fill available
space without needing +100

Finally, there's the code in the '# Burn bootloader' section, which I
think makes sense as a couple new params to the bootloader command,
--uboot and --barebox, assuming these are generic enough that many
images in the future will need to do the same thing.  If not, then we
need to do a similar thing to --source e.g. --type=barebox, etc.

In any case, since those don't relate to any partition, we're going to
have to add a new hook for those, into which you can put the code for
each one of those cases.

So, here's essentially what I think the kickstart syntax would look

part /boot --source bootimg --ondisk sda --fstype=nitrogen6xboot --align 4096
# or:
# part /boot --source nitrogen6xboot --ondisk sda --fstype=nitrogen6xboot --align 4096

part / --source rootfs --ondisk sda --fstype=squashfs --align 8192
part --ondisk sda --size= --fstype=ext4 --align 8192

bootloader  --uboot --timeout=10  --append="rootwait rootfstype=squashfs console=ttyPCH0,115200 console=tty0"
# or:
# bootloader  --type=barebox --timeout=10  --append="rootwait rootfstype=squashfs console=ttyPCH0,115200 console=tty0"


Well, I think that gives us a good foundation for next steps.  What's
your timeline for needing to get this done?  The things I mentioned at
the top are things that I'd need to add framework support for - these
are essentially things that were vaguely known would be required at some
point but I didn't get to in the initial cut and it's good that we have
actual use cases now to flesh things out.  The other things, mentioned
as NOTEs were planned for 1.6 and still are.  Let me know and we can
work something out for getting this done as soon as we can...
Comment 1 Tom Zanussi 2014-02-04 02:25:01 UTC
I just submitted a patchset implementing --source plugins which should help with a lot of this:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/wic-plugins-v0
Comment 2 Tom Zanussi 2014-02-18 16:00:12 UTC
A few more things we need to address that Ross ran into...

> It can't find parted.  On Debian it's in /sbin but normal users
> don't have that in their path.  Can wic attempt to search
> /sbin:/usr/sbin as well as $PATH when looking for it?
> 

We need to add an initial pass that explicitly looks for the tools it needs before proceeding.  We also need to add error checking for the actual invocation of the tools as well - a lot of the original code that's used didn't have this problem as it was part of the build system, but we have a less controled environment.

The assumption until now was that it would at some point just use the native tools from the build, which it can still do, but we need to check that explicitly for the above reason and also for this reason, which is a silent failure with the same root cause:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=5835

> Why does it write the images to /var/tmp/wic?  It's quite likely that
> /var/tmp is actually a tmpfs so that doesn't seem like the best
> choice.  Why can't it use tmp/work tmp/deploy as bitbake does?
> 

This is directly from the underlying 'mic' implementation, which I
didn't see a reason to change... until now...

One other point is that it's supposed to be independent of yocto/oe-core
(though it does use 'bitbake -e' for convenience, but really that should
be viewed as a layering on top of the base raw args version that
explicitly points to the individual artifacts), so really shouldn't know
about tmp/work.  Well, you asked why, which doesn't mean we can't change
it...

We probably need to just make the output dir a command-line option.

> -
> Doesn't work when rm_work is enabled.  It appears to take the rootfs
> from the work directory of the image you specify, instead of the
> tarball or ext3 in tmp/deploy/images, which means when rm_work is
> enabled it can't find the filesystem.  Why doesn't it take content
> from deploy?
> 

Same reason as above, the intention was that it be independent of
oe-core and things manually specified using the -r, etc options.

Actually the real reason is that wic is independent of the image-generation artifacts deployed from the build, those are just there because that's how we typically get the intermediate pieces wic does need.  Also wic needs to be able to generate the rootfs as a user-specified size, which means that won't be the ext3 image created in deploy.  So rm_work is actually removing the pieces we need.
Comment 3 Darren Hart 2014-02-18 17:09:12 UTC
now that fdisk properly supports gpt, we might consider moving to fdisk as parted is such a pain to deal with (missing FS IDs, etc).
Comment 4 Tom Zanussi 2014-02-19 21:00:38 UTC
(In reply to comment #2)
> A few more things we need to address that Ross ran into...
> 
> > It can't find parted.  On Debian it's in /sbin but normal users
> > don't have that in their path.  Can wic attempt to search
> > /sbin:/usr/sbin as well as $PATH when looking for it?
> > 
> 
> We need to add an initial pass that explicitly looks for the tools it needs
> before proceeding.  We also need to add error checking for the actual
> invocation of the tools as well - a lot of the original code that's used
> didn't have this problem as it was part of the build system, but we have a
> less controled environment.
> 
> The assumption until now was that it would at some point just use the native
> tools from the build, which it can still do, but we need to check that
> explicitly for the above reason and also for this reason, which is a silent
> failure with the same root cause:
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5835
> 

BTW, I forgot that we actually do use the native commands, it's just that it didn't actually work sometimes because of a couple bogus PATH elements.

> > Why does it write the images to /var/tmp/wic?  It's quite likely that
> > /var/tmp is actually a tmpfs so that doesn't seem like the best
> > choice.  Why can't it use tmp/work tmp/deploy as bitbake does?
> > 
> 
> This is directly from the underlying 'mic' implementation, which I
> didn't see a reason to change... until now...
> 
> One other point is that it's supposed to be independent of yocto/oe-core
> (though it does use 'bitbake -e' for convenience, but really that should
> be viewed as a layering on top of the base raw args version that
> explicitly points to the individual artifacts), so really shouldn't know
> about tmp/work.  Well, you asked why, which doesn't mean we can't change
> it...
> 
> We probably need to just make the output dir a command-line option.
> 
> > -
> > Doesn't work when rm_work is enabled.  It appears to take the rootfs
> > from the work directory of the image you specify, instead of the
> > tarball or ext3 in tmp/deploy/images, which means when rm_work is
> > enabled it can't find the filesystem.  Why doesn't it take content
> > from deploy?
> > 
> 
> Same reason as above, the intention was that it be independent of
> oe-core and things manually specified using the -r, etc options.
> 
> Actually the real reason is that wic is independent of the image-generation
> artifacts deployed from the build, those are just there because that's how
> we typically get the intermediate pieces wic does need.  Also wic needs to
> be able to generate the rootfs as a user-specified size, which means that
> won't be the ext3 image created in deploy.  So rm_work is actually removing
> the pieces we need.
Comment 5 Tom Zanussi 2014-05-20 16:04:04 UTC
Closing this - more specific work should create new tracking bugs.
Comment 6 Costin 2014-09-29 05:31:44 UTC
As of last comment I will mark this as verified