Bug 13554 - testimage: allow runtime tests to trigger installation of packages
Summary: testimage: allow runtime tests to trigger installation of packages
Status: RESOLVED WONTFIX
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: testimage (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: Future
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-09-25 21:02 UTC by Paul Eggleton
Modified: 2020-02-13 16:39 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2019-09-25 21:02:55 UTC
As the number and complexity of tests increases it gets more difficult to always have an image that contains all of the packages required for testing. If a package feed is available we can of course install the additional packages separately from the tests and then run them, however this is problematic because (a) if you add a required package to a test you then have to separately update the script to install the package, and (b) you then have to hack the manifest so that the package will appear as being present in the image.

Thus, given we already have OEHasPackage() to mark a test with its required packages, it would be useful if testimage were able to automatically install those packages assuming that is possible (i.e. runtime packaging is enabled and there is a feed with the packages in it). Thus the code implementing OEHasPackage() should be changed as follows:

1) Check if runtime packaging is available. The quick and dirty way of doing this would be to check if the package or known path corresponding to the package manager selected is present in the image.

2) If #1 is true, call the appropriate package manager frontend remotely to install the required packages. (NOTE: If this operation fails, the test result should be failure and not skipping - we want to be notified if this happens.)

3) If #1 is not true, proceed with the original logic i.e. checking the manifest and skipping if any of the requested packages are not present in the image.

The "appropriate package manager" would be determined by checking PACKAGE_CLASSES.

We will also want to have a variable controlling this new behaviour e.g. TESTIMAGE_INSTALL_PACKAGES or something like that; if it is not set to "1" then the previous behaviour (#3 above) should apply.
Comment 1 Paul Eggleton 2019-09-25 23:29:09 UTC
One part I missed: it is to be assumed that the configuration of the package manager in the image (i.e. to be able to access the package feed) is done externally, i.e. it's already set up when testimage comes to install the packages.
Comment 2 Richard Purdie 2019-09-26 15:04:57 UTC
I want to state for the record that this adds a ton of extra dimensions to testing and I'm very nervous about testsing automatically attempting "magic" installation of packages. For example does a given image have enough space?

We're struggling as a project to triage and fix problems with the existing tests as nobody is stepping up. Adding these extra dimensions to tests is therefore something which gives me serious worries.
Comment 3 Randy MacLeod 2020-01-16 16:29:02 UTC
Is this going ahead?
Comment 4 Mohamad Noor Alim 2020-01-17 01:53:08 UTC
I am not working on this.
Comment 5 Tim Orling 2020-02-13 16:39:31 UTC
I have to agree with Richard on this one. The motivation and intent is understandable, but this could rapidly turn into a nightmare. The test frameworks that assume they are running on a standard desktop distribution and have access to the internet to download and install new dependencies are...perhaps of questionable design, especially in our context.

A different approach, such as improved discovery of dependencies in the tests themselves (via devtool or build-time) or improved handling of missing run-time dependency reporting (like the error messages for SRC_URI checksums which basically tell you what to paste into recipe) would be more generally useful.