<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>9678</bug_id>
          
          <creation_ts>2016-05-26 20:11:55 +0000</creation_ts>
          <short_desc>Include automated test execution and reporting in Autobuilder</short_desc>
          <delta_ts>2018-05-28 02:07:39 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>5</classification_id>
          <classification>Infrastructure</classification>
          <product>AutoBuilder</product>
          <component>autobuilder</component>
          <version>2.2</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>INVALID</resolution>
          
          
          <bug_file_loc>http://centaurus01.png.intel.com:8010/builders/nightly-testopia/builds/60</bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>enhancement</bug_severity>
          <target_milestone>4.99</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Benjamin Esquivel">benjamin.esquivel</reporter>
          <assigned_to name="Aaron Chan">aaron.chun.yew.chan</assigned_to>
          <cc>aaron.chun.yew.chan</cc>
    
    <cc>apoorv.sangal</cc>
    
    <cc>bill.randle</cc>
    
    <cc>bluelightning</cc>
    
    <cc>ee.peng.yeoh</cc>
    
    <cc>infras.ab.watcher</cc>
    
    <cc>Infras.watcher</cc>
    
    <cc>joshuagloe</cc>
    
    <cc>leonardo.sandoval.gonzalez</cc>
    
    <cc>richard.purdie</cc>
    
    <cc>ross.burton</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>Don&apos;t know</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>62488</commentid>
    <comment_count>0</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2016-05-26 20:11:55 +0000</bug_when>
    <thetext>The purpose of this bug is to add functionality to the AB for executing the automated test suites.

The tests targeted for inclusion are: testimage, oe-selftest, performance measurements, etc.

One requirement is that all the automated tests that QA runs in a release test cycle can be run at the Autobuilder.

For reporting, initially the target will be to report back to the Testopia session that was setup for the QA cycle. Later will be required another bug for reporting to a test report store that is being tracked at bug 9559.

A text test report should exist at the Autobuilder reporting step for the cases where communication to the test report tool fails. There should be a way to see the results at the execution instance which is the AB in this case.

Test selection will be pre-set for this bug (all tests, per-component test AB job separation). Later, another bug needs to take care of being able to select which tests will be run (perhaps via a field in the forced builds page?) 

Anibal has been working on this for weeks now but here&apos;s the bug to track that work.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>63316</commentid>
    <comment_count>1</comment_count>
    <who name="Leonardo Sandoval Gonzalez">leonardo.sandoval.gonzalez</who>
    <bug_when>2016-06-23 19:55:36 +0000</bug_when>
    <thetext>My fault, moving it to M1 again. CCB should agree on any milestone movement.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>63554</commentid>
    <comment_count>2</comment_count>
    <who name="Leonardo Sandoval Gonzalez">leonardo.sandoval.gonzalez</who>
    <bug_when>2016-06-30 20:27:33 +0000</bug_when>
    <thetext>Demo created (http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/) but it needs more work in order to close this bug.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>64065</commentid>
    <comment_count>3</comment_count>
    <who name="Benjamin Esquivel">benjamin.esquivel</who>
    <bug_when>2016-07-15 17:17:18 +0000</bug_when>
    <thetext>This becomes a High if moved to 2.2 M3</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>65517</commentid>
    <comment_count>4</comment_count>
    <who name="Leonardo Sandoval Gonzalez">leonardo.sandoval.gonzalez</who>
    <bug_when>2016-08-25 17:23:11 +0000</bug_when>
    <thetext>This work wont be ready for this release. Moving to next release, expecting more involvement from QA team.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>70163</commentid>
    <comment_count>5</comment_count>
    <who name="Leonardo Sandoval Gonzalez">leonardo.sandoval.gonzalez</who>
    <bug_when>2017-01-25 21:55:05 +0000</bug_when>
    <thetext>Anibal, can you please comment on this one. Ideas to split it?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>71823</commentid>
    <comment_count>6</comment_count>
    <who name="Aníbal Limón">anibal.limon</who>
    <bug_when>2017-03-28 18:00:09 +0000</bug_when>
    <thetext>(In reply to comment #5)
&gt; Anibal, can you please comment on this one. Ideas to split it?

There are two needed tasks for complete this,

1) Finish the testopia_update script.
2) Add the buildstep into the Autobuilder for automatic report (call testopia_update).

Currently Jose Perez done the 1) part so the only remaining part will be add the buildstep.

I&apos;ll change the owner to Jose Perez and now the split isn&apos;t needed.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>73658</commentid>
    <comment_count>7</comment_count>
    <who name="Leonardo Sandoval Gonzalez">leonardo.sandoval.gonzalez</who>
    <bug_when>2017-06-02 19:45:38 +0000</bug_when>
    <thetext>*** Bug 10580 has been marked as a duplicate of this bug. ***</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>79212</commentid>
    <comment_count>8</comment_count>
    <who name="Aaron Chan">aaron.chun.yew.chan</who>
    <bug_when>2018-01-25 04:58:27 +0000</bug_when>
    <thetext>With reference to http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/

Base on the usage model of the testopia_update.py script, it appears that the test cases update is much faster manually compare to updating the test cases result using the script sourcing with the results logfiles. 

In some cases, the results updating results into Testopia appeas to be locking up, unresponsive for several hours and unable to fully complete the result publishing even when running on a bare-metal server. 

Hence the testopia_update.py script require some effort to debug and analysis the issue am facing.

On other areas of enhancement which was tested in autobuilder.

1. Added a patch UploadTestopia.py as an automated buildstep option
2. Modify the buildset-config jobs to adopt the buildstep in item 1.
3. Changes to the job specific interface to accept &quot;Test Run ID&quot; as string 
   parameter to invoke the testopia_update script to upload the auto testcases

Issue Blocking:

1. Testopia_update.py script does not work as per expected
2. Logfiles can be generated in various paths, different logfile names, and 
   some job configuration requires multiple logfiles of various test shares 
   the same test run ID in Testopia.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>80650</commentid>
    <comment_count>9</comment_count>
    <who name="Aaron Chan">aaron.chun.yew.chan</who>
    <bug_when>2018-05-28 02:07:39 +0000</bug_when>
    <thetext>This ticket is back dated 2 years ago and the direction has been moved to
test cases management to a different platform whereby Testopia will be retired replacing by other Test case management tool by the community.

Hence, this ticket is no longer valid and the status is now changed to Never Fix. A new ticket should be raised indicating the same motivation and assign to
the dedicated owner currently working on the enhancements.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>