The shear number of calls to query rpm during rootfs construction, particularly on a large image like core-image-sato-sdk is crazy and badly hurting our performance. The autobuilder struggles due to this as well, its probably one of the major overheads. This is componunded by the number of execs piping the result through grep, cut and so on. Instead of so many calls to rpm -q, we should build a more direct look up cache and use that directly. If we can't do this efficiently in shell, we should write a utility to handle it and call into it. We can likely massively speed up do_rootfs if we fix these things, its not rpm itself which is slowing us down, its the number of exec() calls.
We should instrument the calls to rpm -q, many of these can likely be batched together and provide basic performance enhancements. RPM itself is quite fast as a cache, once the resolver (or install solution) DBs have been generated.
As I said, its not rpm's speed I'm complaing about, its the shear number of rpm calls (hundreds of thousands in the sdk case). The problem is you're doing resolution of packages in loops using resolve_package_rpm (which calls rpm -q). Let me just paraphrase some of package_rpm.bbclass: for pkg in ${package_linguas}; do pkg_name=$(resolve_package_rpm ${confbase}-${archvar}.conf ${ml_pkg}) done for pkg in ${package_to_install} ; do pkg_name=$(resolve_package_rpm ${confbase}-${archvar}.conf ${ml_pkg}) done for pkg in ${package_attemptonly} ; do pkg_name=$(resolve_package_rpm ${confbase}-${archvar}.conf ${ml_pkg}) done followed by the recommendation loop These loops end up involving long lists of packages with many iterations through the execution. resolve_package_rpm also then as a " for solve in `cat ${conffile}`; do" loop which just compounds the problem. To demonstrate the problem, just execute a sato-sdk rootfs generation under set -x or put an echo statement in resolve_package_rpm(). The trouble is we need to pin particular packages down to particular solution databases in resolution order. There has to be a way of doing this more efficently than we currently are though.
The purpose of these blocks of code are to take the OE package names and transform them into a specific filename for the package. The logic is to take the "to-be-installed" package name, check each of the resolver databases until the name is found. Once the initial set of requirements is implemented, all steps after that are automatic with the internal resolver. Do we really have a large number of items specified in the various install variables that this will be an issue? An alternative will be to resolve more then one package at a time (name to filename), however this may add complications as well.
After adding instrumentation, the problem is more clear. It's not the resolution of the initial package installation that is the issue, but instead it's the identification and installation of the optional language files. Once the main install has been resolved, for each package resolves we further look for language files. It would likely be significantly faster if we could batch this search and missing language files are NOT a failure.
The large number of language files is handled by: image.bbclass: rootfs_install_all_locales() This function gets a list of all of the packages installed into the system: INSTALLED_PACKAGES=`list_installed_packages | egrep -v -- ... Then processes these one at a time: for lang in $SPLIT_LINGUAS; do for pkg in $INSTALLED_PACKAGES; do existing_pkg=`rootfs_check_package_exists $pkg-locale-$lang` if [ "$existing_pkg" != "" ]; then PACKAGES_TO_INSTALL="$PACKAGES_TO_INSTALL $existing_pkg" fi done done rootfs_check_package_exists simply calls RPM (or any other package manager) one at a time and asks if a single package exists. This is incredibly inefficient. In order to speed up the install, this function will have to be reworked to provide a larger list of packages to resolve for installation, instead of doing one package at a time. (Also in the case of a very large file system, the package_to_install variable could get -very- large and perhaps exceed the command line length. So refactoring that is likely necessary as well.
Moving to M3 as Mark does not have time for this now. Will take a look at M3 planning again.
FYI, the latter part of this is likely to be fixed by me as part of the work for #2614.
I've sent out some patches to improve this by performing the querying in a batch within a separate C program, however the time saving (with buildhistory disabled) has been offset by other changes which make do_rootfs take longer, so there's probably more work to be done here.
Improvements merged in revision a73c25d2ded3a72159f2ce527e7307808c734686. I'm leaving this open for now in case we can optimise things further.
Since the underlying inefficiency and suggested solution have now been implemented, I've filed enhancement bug 2871 to cover further performance improvements and mark this one as fixed.
(In reply to comment #10) > Since the underlying inefficiency and suggested solution have now been > implemented, I've filed enhancement bug 2871 to cover further performance > improvements and mark this one as fixed. Closing this as Paul suggested.