During discussions with the team, Brian made the call to use the "intermediate" table pattern for storing data for display. As such, we need to create tables that correspond to the objects that appear in the frontend design.

* ProjectAvailableLayer - this lists the layers that are available for adding in the current project; it's a cache of the Layer_Versions table, filtered for the project; it has unique layer names; it points back to the layer which would have to be checked out on the build command.

* the ProjectLayer table (containing the layers currently added in the project) need to reference the ProjectAvailableLayer table instead of the Layer_Version table;

* ProjectAvailableRecipe - this lists the recipes that are available for adding in the current project; it is generated by selecting the Recipe objects available for the current project; it hold entries to the ProjectAvailableLayer that is providing the recipes - as such, the computation of which Recipes are actually build-able is done by selecting the objects linked to ProjectAvailableLayer which are listed in ProjectLayer

* the ProjectAvailableRecipe also holds definitions of custom recipes for this object; the definition names are entries in this table, with the content stored in the ProjectCustomRecipeContent table; at the current moment, the content is thought of package lists, so each entry in the ProjectCustomRecipeContent is a package name, for example

* the list of packages that is available to the Project is another table - ProjectAvailablePackage; which is populated from the Package table, once a build is done; The ProjectAvailablePackage needs to hold a link to ProjectAvailableRecipe (analogous to the Package table containing a link to the  Recipe table) so we can now which package is provided by which Recipe and which Layer.


The particulars of the data fields on the ProjectAvailable* objects are driven by the interface needs. In any case, the *created* and *updated* fields can be automatically populated by Django (see bldcontrol.models)


I think that the most efficient method of keeping these tables in sync with data coming in from the different sources (LayerIndex, Builds, Imported Layers, etc) is to use django.db.models.signals to get hooks on when Layer_Version, Recipe and Package data get created/updated/deleted and perform corresponding actions on the ProjectAvailable* objects.

Alex, thank you for detailed info!

Created attachment 2601 [details]
All Toaster tables

I am not sure this helps at all, but attached is a description of the contents of all Toaster tables.

After looking at the code working with ProjectLayer table I think it's quite dangerous to change it to work with ProjectAvailableLayer instead of Layer_Versions. It looks like a refactoring work, which I'd not recommend to do now considering 1.9 schedule and the current project status.

I'm going to keep all new models as independent from the current functionality as possible. This should allow us to implement image customization without breaking existing functionality.

Opinions?

Just for FYI, I have a patch that generates the correct layer_version entries for the project layers list:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=adamian/20150727_tables&id=4cd4aedf99c942a1cfad4bcad59b6bcf02fd9e98

This enables the correct displaying of the "available layers for project" table information, and it generates what the "ProjectAvailableLayer" table should contain for a specific project.

The performance is not too bad, and if it easier to create Project methods that return querysets for the project available layers, recipes and packages, as an intermediate step to create the caching tables.

Created first 3 models: https://lists.yoctoproject.org/pipermail/toaster/2015-July/002435.html

Please, suggest what to do next. Thank you.

It would be also good to know about possible set of REST APIs required by frontend.

Added CustomImageRecipe model to the poky-contrib:ed/toaster/ic-models. Please review.

Added draft of the ProjectAvailableRecipesTable to the same branch. It requires more work on 'Customise' column. The rest of content is more or less ok.

Added draft code to populate ProjectAvailableLayer and ProjectAvailableRecipe with the data from layer index and builds. This should be enough to start working on frontent of this view: https://www.yoctoproject.org/toaster/all-image-recipes.html

Implemented set of ReST API calls and covered their code with tests: https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ed/toaster/ic-models

Verified as part of 2.1.rc1: 296dfbc9b3a3c0ed64b8e748c2868928b9c2aa8b