<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>13554</bug_id>
          
          <creation_ts>2019-09-25 21:02:55 +0000</creation_ts>
          <short_desc>testimage: allow runtime tests to trigger installation of packages</short_desc>
          <delta_ts>2020-02-13 16:39:31 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>10</classification_id>
          <classification>QA/Testing</classification>
          <product>Runtime Testing</product>
          <component>testimage</component>
          <version>unspecified</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>WONTFIX</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>enhancement</bug_severity>
          <target_milestone>Future</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Paul Eggleton">bluelightning</reporter>
          <assigned_to name="Unassigned">unassigned</assigned_to>
          <cc>apoorv.sangal</cc>
    
    <cc>randy.macleod</cc>
    
    <cc>richard.purdie</cc>
    
    <cc>TicoTimo</cc>
    
    <cc>tim.orling</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Yes (doc changes required)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>85072</commentid>
    <comment_count>0</comment_count>
    <who name="Paul Eggleton">bluelightning</who>
    <bug_when>2019-09-25 21:02:55 +0000</bug_when>
    <thetext>As the number and complexity of tests increases it gets more difficult to always have an image that contains all of the packages required for testing. If a package feed is available we can of course install the additional packages separately from the tests and then run them, however this is problematic because (a) if you add a required package to a test you then have to separately update the script to install the package, and (b) you then have to hack the manifest so that the package will appear as being present in the image.

Thus, given we already have OEHasPackage() to mark a test with its required packages, it would be useful if testimage were able to automatically install those packages assuming that is possible (i.e. runtime packaging is enabled and there is a feed with the packages in it). Thus the code implementing OEHasPackage() should be changed as follows:

1) Check if runtime packaging is available. The quick and dirty way of doing this would be to check if the package or known path corresponding to the package manager selected is present in the image.

2) If #1 is true, call the appropriate package manager frontend remotely to install the required packages. (NOTE: If this operation fails, the test result should be failure and not skipping - we want to be notified if this happens.)

3) If #1 is not true, proceed with the original logic i.e. checking the manifest and skipping if any of the requested packages are not present in the image.

The &quot;appropriate package manager&quot; would be determined by checking PACKAGE_CLASSES.

We will also want to have a variable controlling this new behaviour e.g. TESTIMAGE_INSTALL_PACKAGES or something like that; if it is not set to &quot;1&quot; then the previous behaviour (#3 above) should apply.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85073</commentid>
    <comment_count>1</comment_count>
    <who name="Paul Eggleton">bluelightning</who>
    <bug_when>2019-09-25 23:29:09 +0000</bug_when>
    <thetext>One part I missed: it is to be assumed that the configuration of the package manager in the image (i.e. to be able to access the package feed) is done externally, i.e. it&apos;s already set up when testimage comes to install the packages.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85095</commentid>
    <comment_count>2</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-09-26 15:04:57 +0000</bug_when>
    <thetext>I want to state for the record that this adds a ton of extra dimensions to testing and I&apos;m very nervous about testsing automatically attempting &quot;magic&quot; installation of packages. For example does a given image have enough space?

We&apos;re struggling as a project to triage and fix problems with the existing tests as nobody is stepping up. Adding these extra dimensions to tests is therefore something which gives me serious worries.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>86056</commentid>
    <comment_count>3</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2020-01-16 16:29:02 +0000</bug_when>
    <thetext>Is this going ahead?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>86062</commentid>
    <comment_count>4</comment_count>
    <who name="Mohamad Noor Alim">mohamad.noor.alim.hussin</who>
    <bug_when>2020-01-17 01:53:08 +0000</bug_when>
    <thetext>I am not working on this.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>86435</commentid>
    <comment_count>5</comment_count>
    <who name="Tim Orling">tim.orling</who>
    <bug_when>2020-02-13 16:39:31 +0000</bug_when>
    <thetext>I have to agree with Richard on this one. The motivation and intent is understandable, but this could rapidly turn into a nightmare. The test frameworks that assume they are running on a standard desktop distribution and have access to the internet to download and install new dependencies are...perhaps of questionable design, especially in our context.

A different approach, such as improved discovery of dependencies in the tests themselves (via devtool or build-time) or improved handling of missing run-time dependency reporting (like the error messages for SRC_URI checksums which basically tell you what to paste into recipe) would be more generally useful.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>