When you run a build using shared state and nothing, absolutely nothing changes, Toaster does not collect package information. Before image customisation, this was mostly an annoyance: the build information would be incomplete. With image customisation, however, this becomes a blocker, since you can find yourself in a situation where you keep building a recipe, but the packages for that recipe never show up in the list of packages available to add to your custom images. The attached designed document suggests a workaround for this situation.
Created attachment 2997 [details] Design - Improving handling of builds without package data
Right, so Michael was right: I cannot reproduce this problem across projects within a Toaster instance. The way I encountered it was like this: 1. Set up Toaster and built a few things 2. Deleted the Toaster db 3. Started Toaster on a clean db 4. Built the same things as in 1 This is good news (it's very much an edge case), but it would still be good to get package data from builds that reuse sstate. So hopefully Brian can find a way.
You can also encounter this if the toaster instance has an SSTATE_MIRROR set. Assuming the SSTATE_MIRROR has *all* the sstate you need for a build You can do: 1) Build core-image-minimal This will result in 28 packages available, even though they were populated from the sstate. If you had built toaster you would have had a few hundred available since we let you add host side tools packages to the image. 2) Build dropbear - This will result in *no* additional packages available since it was setscened from sstate and not used in an image. the dropbear<PR>.rpm get's created though. 3) If you customize an image off of core-image-minimal, you will *not* see dropbear available as a package to add. Furthermore you will not see any of the host side tools (like flex) available either which you would if you had built rather than setscened the image. There's a pretty easy fix that will let us "see" packages that are setscened as well as packages that are built. We probably need some other language than "built" to refer to them, though. ??
(In reply to comment #3) > You can also encounter this if the toaster instance has an SSTATE_MIRROR > set. Assuming the SSTATE_MIRROR has *all* the sstate you need for a build > You can do: > 1) Build core-image-minimal > This will result in 28 packages available, even though they were populated > from the sstate. If you had built toaster you would have had a few hundred > available since we let you add host side tools packages to the image. > 2) Build dropbear - This will result in *no* additional packages available > since it was setscened from sstate and not used in an image. the > dropbear<PR>.rpm get's created though. > 3) If you customize an image off of core-image-minimal, you will *not* see > dropbear available as a package to add. Furthermore you will not see any of > the host side tools (like flex) available either which you would if you had > built rather than setscened the image. Thanks for this. So it is a bit worse than I thought .... > There's a pretty easy fix that will let us "see" packages that are setscened > as well as packages that are built. ... luckily, this is good news :) We probably need some other language > than "built" to refer to them, though. ?? Maybe. So that table shows the list of packages generated by the recipes that were needed for the build. How the packages were generated (if the tasks were executed or not, or if they reused shared state objects) is the subject of the "tasks" section. The packages built table does not concern itself with that. It just tells you: ok, these are the packages produced by the recipes bitbake had to "build", independently of how the building was carried out. We picked "packages built" to highlight the difference with "packages included", which are the subset of packages built that ends up installed in an image. We can probably find a better expression than "packages built" (any suggestions?), but in the packages table I would not make any differences in presentation based on how the packages were generated. I don't think users are worried about that when looking at that table. The feedback I've received on the tasks information is that it is too low level to be of much interest in normal circumstances, so if you are looking at the packages list I am not sure you care about how bitbake actually generated the packaged output. You can find out, by looking at the recipe that produced the package, but I don't think many people check that information. Sorry, that was quite long. So, in short: should we change "packages built" and "packages included" to something else?
I would probably just call them packages and leave it at that. How they got there: build locally, setscened from sstate, stork... The how is covered in the build statistics.
(In reply to comment #5) > I would probably just call them packages and leave it at that. That works for me :) How they got > there: build locally, setscened from sstate, stork... The how is covered in > the build statistics.
I can not get the workflow in the design document to actually show an issue, there are two problems. I suspect that this is still required when sstate-cache directories are common across different installations, but I haven't tried that. Still I need a test-case that fails before I can determine if the fixes resolve the issue. Design issue 1 - After creating project proj1 and creating/building customImage-1 based on core-image-minimal, openssl is already in the parsed package list. A better package is atk, which must be built as a software recipe before showing up on the customImage-1 list. Then I can add atk to the image and build. Design issue 2 - After creating new project proj2, and building a new custom image based on core-image-minimal, customImage-2 like the workflow described in the spec, and build core-image-minimal, then the design spec notes: "Since I’ve never built openssl(atk) within this project, I need to build the openssl recipe first". That is not correct, atk already appears as an un-added package on the customImage-2 pages even though it wasn't in the original core-image-minimal package list, not shown on customImage-1, until it was built inside of proj 1. So building the software recipe is NOT required, although oddly building the software recipe does generate the "No Packages Found" message. I'll try using sstate-cache to demonstrate a use case that does require this fix, otherwise the problem may be the collection/display of data from the proj2 atk sofware recipe build's Packages page which is inconsistent with the data shown on the proj2/customImage-2's Add|Remove customimage page.
(In reply to comment #7) If I had scrolled below "Additional Comments" I would have found that Belen and Michael have already observed some of the issues that I've reported - within a toaster instance sstate-cached package's information is available in a 2nd project's customimage page. > I can not get the workflow in the design document to actually show an issue, > there are two problems. I suspect that this is still required when > sstate-cache directories are common across different installations, but I > haven't tried that. > > Still I need a test-case that fails before I can determine if the fixes > resolve the issue. > > Design issue 1 - After creating project proj1 and creating/building > customImage-1 based on core-image-minimal, openssl is already in the parsed > package list. A better package is atk, which must be built as a software > recipe before showing up on the customImage-1 list. Then I can add atk to > the image and build. > > Design issue 2 - After creating new project proj2, and building a new custom > image based on core-image-minimal, customImage-2 like the workflow described > in the spec, and build core-image-minimal, then the design spec notes: > "Since I’ve never built openssl(atk) within this project, I need to build > the openssl recipe first". > That is not correct, atk already appears as an un-added package on the > customImage-2 pages even though it wasn't in the original core-image-minimal > package list, not shown on customImage-1, until it was built inside of proj > 1. > So building the software recipe is NOT required, although oddly building the > software recipe does generate the "No Packages Found" message. > > I'll try using sstate-cache to demonstrate a use case that does require this > fix, otherwise the problem may be the collection/display of data from the > proj2 atk sofware recipe build's Packages page which is inconsistent with > the data shown on the proj2/customImage-2's Add|Remove customimage page.
The easiest way to see the need for the fix posted to the mailing list is to create an sstate-mirror then point a toaster instance at that mirror. I'm a tad confused; what are we trying to replicate exactly?
I think there was some confusion here: the issue was still assigned to Dave Lerner although Brian Avery had already sent a patch to the mailing list. https://lists.yoctoproject.org/pipermail/toaster/2016-March/004078.html Which is already upstream http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118796.html Although not fully sure it has been merged to master yet. Brian's patch probably makes the UI changes unnecessary.
Just found the patch in master :) http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=3f8d5bfdfdcf338a7e7d0925928cf936b7c325aa Setting to resolved - fixed
Verified on master.