I'd like to test toaster against my existing Angstrom setup where all the layers are already there and configured. From what I gleaned from the docs this morning the following issues exist: 1) toaster wants to download layers itself 2) toaster can only handle its own directory layout for layers 3) The .json preconfigures a distro and layer setup
I also care about this use case.
Could you help the developers understand the specific use-case you'd like to see better supported? My inference, based on the subject, is that you'd like to use Toaster against an existing build directory and any existing configuration within (local.conf, bblayers.conf, etc) — is that correct? If so it's worth pointing out that we have partial support for this already through the command-line builds, this setup enables triggering a build through the command-line as usual and monitoring and analysing the build with Toaster.
Set to NEEDINFO whilst we define the requirements.
(In reply to comment #2) > My inference, based on the subject, is that you'd like to use Toaster > against an existing build directory and any existing configuration within > (local.conf, bblayers.conf, etc) — is that correct? That is correct.
In my case, I have a repo manifest customers use to check out various layers (with branch/commit information) and custom template files that define what they should build. This is all done from the command line as our image generation is done on headless machines with automagical scripts. We would like to setup toaster to run against our specific layer/distro configuration so people can easily create custom images specific to their application.
(In reply to comment #5) > In my case, I have a repo manifest customers use to check out various layers > (with branch/commit information) and custom template files that define what > they should build. This is all done from the command line as our image > generation is done on headless machines with automagical scripts. > > We would like to setup toaster to run against our specific layer/distro > configuration so people can easily create custom images specific to their > application. I think this should already be possible to some extent using a custom toasterconf.json file http://cgit.openembedded.org/openembedded-core/tree/meta/conf/toasterconf.json In the config section, you would specify your DISTRO, MACHINE and I guess in your case you might also want to append to DISTRO_FEATURES "config": { "MACHINE" : "ettus-e1xx", "DISTRO" : "nodistro", "DL_DIR" : "${TOPDIR}/../downloads", "IMAGE_FSTYPES": "ext3 jffs2 tar.bz2", "IMAGE_INSTALL_append": "", "PACKAGE_CLASSES": "package_rpm", "SSTATE_DIR" : "${TOPDIR}/../sstate-cache", "DISTRO_FEATURES_append" : " opengl x11" }, In the "bitbake" section, leave only the bitbake version(s) you want your users to use for the builds. Do the same in the "releases" section (or create a new one as needed). Also in the "releases" section, make sure you specify the layers required for your builds as the "defaultlayers". So, if you want Toaster users to build with the master branch for some reason "releases": [ { "name": "master", "description": "OpenEmbedded master", "bitbake": "master", "branch": "master", "defaultlayers": [ "openembedded-core", "e100-bsp", "meta-sdr", "meta-filesystems", "meta-networking", "meta-python" ], "layersourcepriority": { "Imported layers": 99, "Local OpenEmbedded" : 10, "OpenEmbedded" : 0 }, "helptext": "Toaster will run your builds using the tip of the <a href=\"http://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded master</a> branch, where active development takes place. This is not a stable branch, so your builds might not work as expected." }, The layers listed as "defaultlayers" will be automatically added to projects when you create them (i.e. they will be added to bblayers.conf). Machine, distro and distro_features will be also set, which means users just need to select an image/recipe and build. If I am not mistaken (I could very well be) and *in theory* (I can't state this enough) this should work. I have never tried it though. But this is a use case we always knew we'd need to support, so this is a good time to fix it if it's not working, or to start thinking about how we could improve this. Another alternative for when you have tons of layers, which I briefly mentioned to Koen, would be to replace the OpenEmbedded layer index with your own layer index instance, and set it up in the "layersources" section { "name": "OpenEmbedded", "sourcetype": "layerindex", "apiurl": "http://layers.openembedded.org/layerindex/api/", "branches": ["master", "jethro", "krogoth"] }, Another one that should work in theory, but nobody has tried yet.
The point of this bug is to *stop* tools from touching any .conf file, it should use the provides bblayers.conf, local.conf, etc. Touching any DISTRO var is a big no-no as well. If toaster does any of that, it wouldn't be the DISTRO you set anymore since toaster went in and changed things.
(In reply to comment #7) I suspect I might be missing some context about how you want to use Toaster, so I am going to ask some questions. Remember, I am not trying to be recalcitrant. I am just trying to understand what you are asking, which is not necessarily obvious to me. Unfortunately, I have no direct line to what's going on inside your head (that would be a handy thing to have). > The point of this bug is to *stop* tools from touching any .conf file, it > should use the provides bblayers.conf, local.conf, etc. What's the purpose of using Toaster in this scenario? Just collect and display information about your builds? Or do you want other people to run builds with a configuration you provide without being able to change anything? Touching any DISTRO > var is a big no-no as well. If toaster does any of that, it wouldn't be the > DISTRO you set anymore since toaster went in and changed things. Very true, but Toaster right now is partly a configuration tool. The purpose is to make it easier for people to configure their builds. To be that, it needs to be able to change "things". If you don't want Toaster to touch your existing configuration, and you want it only to gather information about your builds, right now you need to start your builds from command line instead of Toaster. We could potentially look into changing that. If you just don't want your Toaster users to change anything in the configuration you provide them, we need to give you a way to "hide" the configuration functionality of Toaster. Bringing in a permissions system might be a way of doing this.
(In reply to comment #8) > (In reply to comment #7) > > I suspect I might be missing some context about how you want to use Toaster, > so I am going to ask some questions. Remember, I am not trying to be > recalcitrant. I am just trying to understand what you are asking, which is > not necessarily obvious to me. Unfortunately, I have no direct line to > what's going on inside your head (that would be a handy thing to have). > > > The point of this bug is to *stop* tools from touching any .conf file, it > > should use the provides bblayers.conf, local.conf, etc. > > What's the purpose of using Toaster in this scenario? Angstrom used to have the narcissus tool, which is as web frontend for 'opkg install'. You selected the MACHINE you want, select package(-groups) and click 'build' to receive a rootfs image. Due to excessive package (not recipe) renaming in OE-core every release this is a pain to keep working. Enter hob and toaster :) So that's where I'm coming from and would like to see, more details below. > Just collect and > display information about your builds? Or do you want other people to run > builds with a configuration you provide without being able to change > anything? Kinda. Changing MACHINE and fiddling with image content is what I'm after. Having all the info and statistics toaster provides is a big win as well, but I can get most of what I want with buildhistory.bbclass right now. The underlying issue is that people suck at creating layers, a lot of manual tweaking to bblayers.conf is needed to get things to work properly (e.g. the xf86 config recipe needs a bbappend in each BSP layer and uses immediate expansion the wrong way). Issues are getting fixed at a glacial pace because it involves a lot of different maintainers, so for the foreseeable future toaster would need to respect an existing bblayers.conf instead of (re)generating it. > Touching any DISTRO > > var is a big no-no as well. If toaster does any of that, it wouldn't be the > > DISTRO you set anymore since toaster went in and changed things. > > Very true, but Toaster right now is partly a configuration tool. The purpose > is to make it easier for people to configure their builds. To be that, it > needs to be able to change "things". Right, but I want it only to change non-DISTRO "things". I realize that is not a well defined thing :) > If you don't want Toaster to touch your existing configuration, and you want > it only to gather information about your builds, right now you need to start > your builds from command line instead of Toaster. We could potentially look > into changing that. > If you just don't want your Toaster users to change anything in the > configuration you provide them, we need to give you a way to "hide" the > configuration functionality of Toaster. Bringing in a permissions system > might be a way of doing this. I did have a hob bug for such a 'kiosk' mode. That had a lot of milestone moving activity ;)
Well, thanks for that: now it makes sense :) There seem to be 2 parts to what you need: 1) A Toaster UI without some of the current options (as shown in the attached document). This looks to me like the easy part. It could probably be enabled via user types and permissions. That would make this feature dependent on bug 6233, which doesn't look like will be done in 2.2 (unless someone volunteers). I wish I had better news: I will set both to 2.3. Now, the resulting Toaster workflow might not be ideal: a more step by step approach, like you probably had in narcissus, I think would suit your use case better. In Toaster we moved away from that kind of design approach because we didn't want to impose a linear workflow on people. That makes sense when you have access to all the Toaster configuration options, but makes less sense when you only have access to a restricted set of option (like in your case). 2) A Toaster that plays nice with your existing bblayers.conf. Toaster currently appends to bblayers.conf a line like this BBLAYERS = "/home/belen/master-test/_toaster_clones/_git___git.yoctoproject.org_meta-yocto_jethro/meta-yocto /home/belen/master-test/_toaster_clones/_git___git.yoctoproject.org_meta-yocto_jethro/meta-yocto-bsp /home/belen/master-test/_toaster_clones/_git___git.yoctoproject.org_poky_jethro/meta /home/belen/master-test/build/toaster-custom-images" To add the location of the layer clones Toaster will use for the build. If this is a problem for your use case, then some additional changes will be needed to the way Toaster runs builds. Sorry, I know this is long, but we need to document these things or we'll forget about them.
Created attachment 3138 [details] Design - Changing only "non-DISTRO" things
(In reply to comment #10) > Well, thanks for that: now it makes sense :) > > There seem to be 2 parts to what you need: > > 1) A Toaster UI without some of the current options (as shown in the > attached document). The changes mentioned in the pdf look good to me. > This looks to me like the easy part. It could probably > be enabled via user types and permissions. That would make this feature > dependent on bug 6233, which doesn't look like will be done in 2.2 (unless > someone volunteers). I wish I had better news: I will set both to 2.3. > > Now, the resulting Toaster workflow might not be ideal: a more step by step > approach, like you probably had in narcissus, I think would suit your use > case better. In Toaster we moved away from that kind of design approach > because we didn't want to impose a linear workflow on people. That makes > sense when you have access to all the Toaster configuration options, but > makes less sense when you only have access to a restricted set of option > (like in your case). I have no strong opinion on that. The narcissus flow was made linear to cope with people who had no understanding of the system. I'm not sure that is true for the people I want to aim toaster that. > 2) A Toaster that plays nice with your existing bblayers.conf. Toaster > currently appends to bblayers.conf a line like this > > BBLAYERS = > "/home/belen/master-test/_toaster_clones/_git___git.yoctoproject.org_meta- > yocto_jethro/meta-yocto > /home/belen/master-test/_toaster_clones/_git___git.yoctoproject.org_meta- > yocto_jethro/meta-yocto-bsp > /home/belen/master-test/_toaster_clones/_git___git.yoctoproject. > org_poky_jethro/meta /home/belen/master-test/build/toaster-custom-images" > > To add the location of the layer clones Toaster will use for the build. If > this is a problem for your use case, then some additional changes will be > needed to the way Toaster runs builds. I have no problems with adding layers that host the custom images/packagegroups/etc. I do have problems with adding distro layers like meta-yocto. > Sorry, I know this is long, but we need to document these things or we'll > forget about them. No problem!
Michael is reviewing since it may already be fixed.
We've made some recent changes which should make it possible to distribute toaster with a custom setup. Toaster can now have layers added to it just by referencing the directories on the file system. These can be imported using the UI or can be provided at set up time. https://wiki.yoctoproject.org/wiki/Toaster_custom_data Also the mechanism in which Toaster's default configuration for oe-core + bitbake is much improved. I'm going to close this bug as a lot of what is discussed is possible now either through providing custom data/settings to Toaster or using the command line builds option. Please feel free to open a new bug with any specific feature request (preferably one per feature).