Bug 6879

Enhancements should not be in M4, but this is a 'design' one, just for design work. No code to be committed will come out of resolving this enhancement.

*** Bug 7250 has been marked as a duplicate of this bug. ***

Created attachment 2487 [details]
Design - Image customisation in Toaster.pdf

This document is the outcome of the preliminary design discussions.

Image recipe customisation is our main deliverable for the 1.9 release. Design must be ready by the end of M1.

The document looks solid and clear to me. Thank you!

Couple of questions and notices:
- As I understood it's quite hard to guess which packages included into the image before building it. Does it make sense to have a requirement that image should be built *before* it can be customised? I mean there should *always* be 'base' image and it has to be built before we can customise it. This would simplify the UI and implementation from my point of view.

- If we don't support package groups it would be quite hard to customize big images. They can contain hundreds or even thousands of packages, which is quite hard to navigate. We need to come up with some grouping solution anyway.

- A lot of projects have quite complex partitioned images. Would it make sense to at least design customisation for partitioned images?

BTW, partitioned images are going to be produced by build system the same way as other types of images. Please, look at the Bug 7672 for details.

Thanks for the comments. Some answers below.

(In reply to comment #5)
> The document looks solid and clear to me. Thank you!
> 
> Couple of questions and notices:
> - As I understood it's quite hard to guess which packages included into the
> image before building it. Does it make sense to have a requirement that
> image should be built *before* it can be customised? I mean there should
> *always* be 'base' image and it has to be built before we can customise it.
> This would simplify the UI and implementation from my point of view.

That's the way Hob works. The result is an interface that feels like is stopping you at every point, and continuously forcing you to wait. We will provide users the option to build the image if they want 100% accurate information about package content and size. But if the information they have through the parsing process is good enough, they can proceed. Hopefully that provides a bit more freedom. 

In reality, you are quite likely to build the image you will choose as a base beforehand, so chances are that lots of times Toaster will have all the information already and this will not be a problem.

> 
> - If we don't support package groups it would be quite hard to customize big
> images. They can contain hundreds or even thousands of packages, which is
> quite hard to navigate. We need to come up with some grouping solution
> anyway.

Definitely: at some point we will have to tackle this, as we mention in page 11 of the document. There are a couple of issues connected to package groups:

1. Currently package groups look to me quite implementation-oriented. We might consider adding new package groups that map better to user goals (I want LAMP in my image kind of thing)

2. We need to expose their content, so that people can remove packages from them

3. Package groups only might not be enough, and we might also consider a package "ontology" of some sort (similar to recipes and their sections)

> 
> - A lot of projects have quite complex partitioned images. Would it make
> sense to at least design customisation for partitioned images?
> 
> BTW, partitioned images are going to be produced by build system the same
> way as other types of images. Please, look at the Bug 7672 for details.

If I am understanding this correctly, the interaction between wic and bitbake is being enhanced to allow you to set wic options through the meta-data. If this is the case, once that's in place, we can definitely start thinking about how to expose that functionality in Toaster. Off the top of my head, we could have an extra step in the image customisation process where you can set IMAGE_FSTYPES and choose from the <wic image> options provided by the base image recipe. 

The interesting bit is: what happens next? Can we run wic from Toaster using the output of the build, so that the final outcome from your interaction with Toaster is a file ready to go into an sdcard and into your board?

Thank you for the explanations! I think we're on the same page regarding 2 first items (base image recipe and package groups).

Let me answer your questions and give more explanations about partitioned images.

> If I am understanding this correctly, the interaction between wic and bitbake is being enhanced to allow you to set wic options through the meta-data.

Not exactly. Most of wic configuration will be in .wks files in the same location as bitbake image recipes. In meta*/recipes*/images/ to be exact.

Only one new option will be added to bitbake image recipes: WIC_WKSFILES. This is a list of .wks files to run wic with.

IMAGE_FSTYPES will be extended. If it contains wic bitbake will run wic with every .wks from WIC_WKSFILES producing partitioned image for every .wks

Bitbake can produce wic images and other type of images (ext3 jffs2 tar.bz2) at the same time as for bitbake 'wic' is just one more type of image. For example if IMAGE_FSTYPES ?= "jffs2 tar.bz2 wic" bitbake will produce 3 types of images. The only difference is that there can be more than one wic image depending on the number of .wks files mentioned in WIC_WKSFILES. 

> If this is the case, once that's in place, we can definitely start thinking about how to expose that functionality in Toaster. Off the top of my head, we could have an extra step in the image customisation process where you can set IMAGE_FSTYPES and choose from the <wic image> options provided by the base image recipe. 

From my point of view user should be able to add/modify .wks files either in plain editor-like mode or in more sophisticated partition editor similar to fdisk.

> The interesting bit is: what happens next? Can we run wic from Toaster using the output of the build, so that the final outcome from your interaction with Toaster is a file ready to go into an sdcard and into your board?

There is no need to run wic from toaster. Bitbake will run wic and produce ready to flash images according to the IMAGE_FSTYPES, WIC_WKSFILES and set of .wks files in meta*/recipes*/images/.

From user point of view the procedure will remain the same - toaster  runs bitbake and bitbake produces images. The way it produces them doesn't actually matter. Either it produces them itself or runs wic - the result(images) will be available for user when bitbake finishes the build.

Thanks, Ed. 

If I understand correctly, once what you describe is in place, you will be able to generate wic images from Toaster as is, since you can currently set any variables you want through it. So you will be able to select 'wic' as part of the IMAGE_FSTYPES value, and you will be able to set WIC_WKSFILES. I've attached an screenshot to give you an idea of how it would look like (wic_variables.png).

Now, this is surely not good enough: at some point we should create a more custom interface that exposes the wks files available and explains the outcome of each, so that users don't need to understand the variables involved or the wks files.

Created attachment 2523 [details]
wic_variables.png

(In reply to comment #8)
> Thanks, Ed. 
> 
> If I understand correctly, once what you describe is in place, you will be
> able to generate wic images from Toaster as is, since you can currently set
> any variables you want through it. So you will be able to select 'wic' as
> part of the IMAGE_FSTYPES value, and you will be able to set WIC_WKSFILES.
> I've attached an screenshot to give you an idea of how it would look like
> (wic_variables.png).

There should be possibility to add .wks files and edit them, I think. That is a minimal functionality we need to support customisation of wic images.

> Now, this is surely not good enough: at some point we should create a more
> custom interface that exposes the wks files available and explains the
> outcome of each, so that users don't need to understand the variables
> involved or the wks files.

Agree.

> There should be possibility to add .wks files and edit them, I think.

This might be a bit harder. The only way of 'adding' things to Toaster right now is by making them part of a layer, and importing that layer into Toaster. 

Toaster does not currently allow you to edit any files (as in it doesn't provide an editor). It uses forms to collect user input. 

But of course everything is open for discussion :)

Design is now done. A high level explanation can be found in this video

https://wiki.yoctoproject.org/wiki/images/9/90/Image-customisation-v2.webm

Detailed design specifications are attached to

bug 8039
bug 8070
bug 8081
bug 8082
bug 8091
bug 8103
bug 8104
bug 8117
bug 8128
bug 8132

Design is done.
Put this on verified.