12654 2018-04-04 19:05:08 +0000 QA test results management to replace Testopia 2019-02-22 01:04:41 +0000 1 1 10 QA/Testing Test Plans/Suite test-plans/suite 5.99 x86 Multiple RESOLVED FIXED High enhancement 2.7 M3 12651 13012 1 ee.peng.yeoh ee.peng.yeoh randy.macleod stephano --- --- No (bug/feature does not impact docs) oldest_to_newest 80105 0 ee.peng.yeoh 2018-04-04 19:05:08 +0000 Objective: To develop automated script to store oeqa testcase execution result and log, then to compare test result and log for 2 different QA cycle Flow: #1: Create test case to test environment matrix for test execution planning #2: Execute testing (handle by AB or manual human execution) #3: Merge test result & log from the execution at #2 to test case environment matrix created at #1 #4: Compare test result & log between 2 QA cycle 80107 1 ee.peng.yeoh 2018-04-04 19:10:16 +0000 Developing for #1: Completed: 1. Developed the script to parse oeqa testcases to capture test case name 2. Developed the script to define test result and log data structure for each test case captured above 3. Developed the script to output the test result and log data structure to json format file In progress: 1. Integrate with oe-git-archive to store test result and log data to git repository 80108 2 ee.peng.yeoh 2018-04-04 19:13:04 +0000 Developing for #1: Completed: 1. Developed the script to parse oeqa testcases to capture test case name 2. Developed the script to define test result and log data structure for each test case captured above 3. Developed the script to output the test result and log data structure to json format file In progress: 1. Integrate with oe-git-archive to store test result and log data to git repository 80355 3 ee.peng.yeoh 2018-04-20 02:32:18 +0000 Completed development for create and update. The “create” will create the test plan for a particular test component (eg. runtime), create the directory structure based on the test environments (eg. os type, board, host machine) defined by user, create the test result json template file, finally store these directory and files into local git repository. The “update” will update test result status (Pass/Fail) from a test log file created from executing oeqa (eg. runtime) into the local git repository created in “create”. 82057 4 ee.peng.yeoh 2018-11-08 06:13:35 +0000 OEQA test result logging http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156990.html http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156991.html http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156992.html http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156993.html 82058 5 ee.peng.yeoh 2018-11-08 06:14:37 +0000 Test result logging storage http://lists.openembedded.org/pipermail/openembedded-core/2018-November/157403.html 82229 6 ee.peng.yeoh 2018-11-15 01:40:21 +0000 Richard shared his ideas on test result managements, where the results need to be collected from the autobuilder, and analysed. Currently, OEQA automated test will output testresults.json to LOG_DIR, proposing a script to store the multiple testresults.json to a common git repository. Currently this script served as the single entry point to result managements, including storage, analyzing, reporting, & manual test execution. Initial patch submitted as below. http://lists.openembedded.org/pipermail/openembedded-core/2018-November/157403.html 82230 7 ee.peng.yeoh 2018-11-15 01:43:28 +0000 Bugzilla ticket below was for developing the script as a plugin to the main test-case-mgmt script, where this new plugin will handle manual test execution based on manual test cases available inside OEQA/manual, and then output test results in the standard testresults.json format. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12651 82833 8 ee.peng.yeoh 2019-01-22 09:52:23 +0000 Revised the "store" and "report" codebase. http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278238.html 82897 9 ee.peng.yeoh 2019-01-30 03:15:40 +0000 Testopia main roles was to store test result and provide data for the test summary report. The resultstool scripts were developed to provide the "store" (storing test results into git repository) and "report" (test summary report). Patches submitted to enable "store" and "report" that will replace testopia. http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278487.html