Objective: To develop automated script to store oeqa testcase execution result and log, then to compare test result and log for 2 different QA cycle Flow: #1: Create test case to test environment matrix for test execution planning #2: Execute testing (handle by AB or manual human execution) #3: Merge test result & log from the execution at #2 to test case environment matrix created at #1 #4: Compare test result & log between 2 QA cycle
Developing for #1: Completed: 1. Developed the script to parse oeqa testcases to capture test case name 2. Developed the script to define test result and log data structure for each test case captured above 3. Developed the script to output the test result and log data structure to json format file In progress: 1. Integrate with oe-git-archive to store test result and log data to git repository
Completed development for create and update. The “create” will create the test plan for a particular test component (eg. runtime), create the directory structure based on the test environments (eg. os type, board, host machine) defined by user, create the test result json template file, finally store these directory and files into local git repository. The “update” will update test result status (Pass/Fail) from a test log file created from executing oeqa (eg. runtime) into the local git repository created in “create”.
OEQA test result logging http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156990.html http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156991.html http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156992.html http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156993.html
Test result logging storage http://lists.openembedded.org/pipermail/openembedded-core/2018-November/157403.html
Richard shared his ideas on test result managements, where the results need to be collected from the autobuilder, and analysed. Currently, OEQA automated test will output testresults.json to LOG_DIR, proposing a script to store the multiple testresults.json to a common git repository. Currently this script served as the single entry point to result managements, including storage, analyzing, reporting, & manual test execution. Initial patch submitted as below. http://lists.openembedded.org/pipermail/openembedded-core/2018-November/157403.html
Bugzilla ticket below was for developing the script as a plugin to the main test-case-mgmt script, where this new plugin will handle manual test execution based on manual test cases available inside OEQA/manual, and then output test results in the standard testresults.json format. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12651
Revised the "store" and "report" codebase. http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278238.html
Testopia main roles was to store test result and provide data for the test summary report. The resultstool scripts were developed to provide the "store" (storing test results into git repository) and "report" (test summary report). Patches submitted to enable "store" and "report" that will replace testopia. http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278487.html