Created attachment 2983 [details] wrong-package-filter-number.png Once again, I am having trouble to reproduce this reliable. But it seems as if, some times, the list of packages added to a custom image returned by the 'added packages' filter is incorrect. In the attached sreenshot, the number of packages added to the custom image I've called 'belen-sato-dev' seems to be 1223. That's what the build data, the custom image page and the right column in the custom image details page say. But the added packages filter says the number is 572.
Probably because the queryset used to set that number on the filter doesn't correctly mirror the querysets used to populate those other pages.
To reproduce this reliably: 1. create a custom image from core-image-minimal (it has 29 packages) 2. add the bash package to it Note that in the popup, the dependency ncurses-libtinfo has empty brackets following it, which suggests there is no size info for it (perhaps it is a package group?). Note that the ncurses-libtinfo package doesn't show up as a package in the list of available packages. The page reports that 2 packages were added, and the page shows that the custom image now has 31 packages; but the "Add | Remove" filter shows 30 packages in the image (so one package is being missed). If you now remove bash, the number of packages in the image drops to 30 (on the page) and 29 (on the filter). This suggests that the ncurses-libtinfo package hasn't been removed, as far as the page is concerned; but does seem to be treated correctly by the filter.
After more investigation, I've found that ncurses-libtinfo is included in the count shown in the "About" panel for a custom recipe (for the situation above), but not included in the packages shown in the ToasterTable on the same page. This is because ncurses-libtinfo has no associated Recipe object, which means it is filtered out of the list of packages which can be added or removed via the table; but it is not filtered out in the list of packages used to give the count for the "About" panel (which is why the latter is greater by 1). So there is a discrepancy in the sources of data for the two parts of the page, but it's not obvious how to fix it. The question is whether ncurses-libtinfo and other packages without a Recipe should be excluded from the "About" count; if so, they should also be removed from the "Add package" dialog, where packages without Recipe objects are currently shown as dependencies. They don't seem to show up in the "Remove package" dialog, but they should be removed from there, too, if they do show up. Dave Lerner is working on bug 9121, which is related.
I'm taking a look at this, bear with me whilst I get to grips with toaster.
I'm currently unable to reproduce this, I have a new toaster setup and created a custom image based on core-image-minimal. Once the initial image was built I built the bash recipe. Once done I add the bash package to my custom image and the about box shows "Approx. packages included 31", filtering the table by packages in my image shows "Add|Remove packages (31)" in the table title → no discrepancy here. Am I missing some detail?
I wonder whether the root cause of this is the issue reported in #9142 "ca-certificates have non ascii characters in filenames which causes toaster to be unhappy". The non-ascii character in one of the certificate filenames results in an exception being thrown whilst handling the ImagePkgList MetadataEvent, presumably resulting in incomplete data being written to the database.
I will try to reproduce again.
I added bash to a custom image before I'd built bash, which still produces the issue I noted in my original comment. I also built bash, then created another custom image and added bash to it, with the same issue. I wondered whether this might be an issue with the order in which I was building stuff, so I cleaned everything out and did: * build bash * build core-image-minimal * create custom image based on core-image-minimal * added then removed the bash package I still see the same issue here, too.
(In reply to comment #8) > I wondered whether this might be an issue with the order in which I was > building stuff, so I cleaned everything out and did: In order to try and reproduce this correctly can you elaborate on what you mean by cleaned everything out here? > * build bash > * build core-image-minimal > * create custom image based on core-image-minimal Packages in image (filter): 29 Approx packages includes (rhs box): 29 > * added then removed the bash package add package: 2 packages added Packages in image (filter): 30 Approx packages includes (rhs box): 31 remove package: 1 package removed Packages in image (filter): 30 Approx packages includes (rhs box): 29 > I still see the same issue here, too. Yes, I can reproduce it now too — thanks for providing reproduction steps. I was under the impression that toaster only let you add packages to an image, not recipes which haven't yet been built. Is my understanding incorrect here? My 'Add | Remove packages' header on the page for my custom image lists 665 packages available but if I navigate to the 'Software Recipes' page and 'Filter by project recipes' I see 'Recipes provided by layers added to this project (968)'
My procedure for "cleaning out" usually means removing tmp/, buildhistory/, and cache/. If I can't reproduce something, I'll sometimes also delete tmp-glibc/, downloads/ and sstate-cache/, just in case, though nowadays I rarely feel the need to do this.
(In reply to comment #2) > To reproduce this reliably: > > 1. create a custom image from core-image-minimal (it has 29 packages) > 2. add the bash package to it > > Note that in the popup, the dependency ncurses-libtinfo has empty brackets > following it, which suggests there is no size info for it (perhaps it is a > package group?). Note that the ncurses-libtinfo package doesn't show up as a > package in the list of available packages. When reproducing this I found it interesting that ncurses-libtinfo has empty brackets following it (indicating no size information is known) even though bash and its dependencies have already been built. I looked at the build info page for the bash build and in packages there isn't any ncurses-libtinfo. Nor is there an ncurses-libtinfo in the packages for the ncurses recipe if I look at the packages tab of the recipes page for ncurses. I guess that's because the debian renaming renames the file to libtinfo5, however there's no libtinfo5 in the packages list either. There is however a package generated in the deploy directory: /home/joshuagl/Projects/poky/build-toaster-2/tmp/deploy/rpm/i586/libtinfo5-6.0+20160213-r0.i586.rpm At this point I figure I need to better understand how the database is populated and why libtinfo doesn't appear anywhere in my packages list for a build which clearly generated it.
After much wading through code I found that the package info dumping hooks in toaster.bbclass were only sending data for the packages explicitly defined in PACKAGES. This doesn't account for dynamically generated subpackages, such as plugins and modules, created via the do_split_packages method of package.bbclass. ncurses uses this to split out the libraries it creates into separate packages which explains Elliot's reproduction steps. In a sato image there are several recipes that make use of this technique, including i.e. gstreamer, which may account for a much larger variance in the packages as show in the attached screenshot. I've written a patch that changes toaster.bbclass to use os.walk on PKGDESTWORK to find and read each of the package data files that have been written, this both simplifies the code (the same method works for do_package and do_packagedata_setscene) and is a little more robust. Following the reproduction steps from earlier: (In reply to comment #8) > * build bash > * build core-image-minimal > * create custom image based on core-image-minimal Packages in image (filter): 29 Approx packages includes (rhs box): 29 > * added then removed the bash package The bash dependencies pop-over shows package size: "ncurses-libtinfo (131.6 KB)" add package: 2 packages added Packages in image (filter): 31 Approx packages includes (rhs box): 31 remove package: 1 package removed Packages in image (filter): 30 Approx packages includes (rhs box): 30
Patch submitted to the toaster list: https://lists.yoctoproject.org/pipermail/toaster/2016-April/004438.html
The fix for toaster.bbclass has been merged into the master branch of openembedded-core and poky: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=f5033175ef51cf234ab5a85293a30bd9d7440d75 Please do update (and reopen) this bug if the issue persists.
I am going to reopen this, because I think what I am coming across might be the same problem (if it isn't, let me know and I'll open a different bug). With master:ee25d0e3987d7732a2e46e1640693b4cf419a9fc I can still see problems with the number and list of packages for custom images shown by Toaster. This is what I did: 1. I created a "master" project, built core-image-base and core-image-minimal 2. Created custom images based on those 2 base images 3. Added and removed a few packages, rebuilding after each step (between them, tcpdump from meta-networking, and apache2 from meta-webserver) None of the above caused any problems. But then 4. I created a second project, this time using "jethro" as the release 5. I created a custom image based on core-image-minimal 6. I built my custom image to get a list of packages And when I went to the custom image details page, I saw what it's shown in incorrect-package-list.png. The right hand column reports 29 packages, but only 1 is shown in the table. I've also attached the database, just in case it helps. I wonder if it happens only when the second project uses the jethro release, or if it happens as well if you choose master or krogoth.
Created attachment 3135 [details] incorrect-package-list.png
(In reply to comment #15) > I've also attached the database, just in case it helps. > No I didn't: way too big to upload. If someone wants to look at it, let me know
(In reply to comment #15) > 4. I created a second project, this time using "jethro" as the release Aha! Here's the problem. The fix was to toaster.bbclass which is part of the OE-Core meta layer which is cloned by Toaster when performing a build. When you create a new project using jethro we clone a branch of jethro's OE-Core meta layer and don't have this fix. So I'm inclined to think the bug is fixed, but now we get to debate over whether the fix should be backported to jethro. > I wonder if it happens only when the second project uses the jethro release, > or if it happens as well if you choose master or krogoth. Based on your steps above I'd think this should be resolved on krogoth & master, broken everywhere else (though I think jethro is the only other release we support in toaster atm?).
Created attachment 3136 [details] krogoth-custom-image.png That explanation makes sense. However, I think I might have reproduced the problem in both a Krogoth project (see attached krogoth-custom-image.png) and a master project (see master-custom-image.png).
Created attachment 3137 [details] master-custom-image.png
Unfortunately I don't have any cycles to work on this in 2.3 so resetting to New and default assignee. We need to try and reproduce per Belén's comments and see if this issue still affects Toaster master.
I still need to try test to see if this is already fixed, but in the meantime I am going to move it out of YP-2.4.
Bulk move from 4.99 or 0.00 to 5.99