Bug 12651

The script should:

1. step through all test steps
2. prompt tester to provide test results (Passed/ Failed)
3. implement check to check tests that has been executed
    a. if executed (with results), skip
4. populate all test results into separate file with json format

Prototype implementation Plan [Status]:

1. Script able to step through all test steps
   [Done]
2. User able to select test case to run
   [New]
3. User able to select a range of test case to run
   [New]
4. Test result prompt should have status: Passed, Failed, Skipped, Blocked.
   [90%, need to add 'BLOCKED' status]
5. Implement a check mechanism to check tests that has been executed
    a. If executed (with results), skip
   [New]
6. Script able to populate all test results into a new json file.
   [80%, debugging to make sure test results are populated in nested dictionary format]

Prototype implementation Plan [Status]:

1. Script able to step through all test steps
   [Done]
2. User able to select test case to run
   [New -> Start to looking this functionality]
3. User able to select a range of test case to run
   [New]
4. Test result prompt should have status: Passed, Failed, Skipped, Blocked.
   [Done]
5. Implement a check mechanism to check tests that has been executed
    a. If executed (with results), skip
   [New]
6. Script able to populate all test results into a new json file.
   [Done]

Work archived in GitHub:
https://github.com/rebeccasf/openembedded-core/blob/devel/meta/lib/oeqa/manual/manual-test-helper

Prototype implementation Plan [Status]:

1. Script able to step through all test steps
   [Done]
2. User able to select test case to run
   [WW17.1: Done]
3. User able to select a range of test case to run
   [WW17.1: Start to look into this]
4. Test result prompt should have status: Passed, Failed, Skipped, Blocked.
   [Done]
5. Implement a check mechanism to check tests that has been executed
    a. If executed (with results), skip
   [New]
6. Script able to populate all test results into a new json file.
   [Done]

Work archived in GitHub:
https://github.com/rebeccasf/openembedded-core/blob/devel/scripts/manual-test-helper

Prototype implementation Plan [Status]:

1. [Done] Script able to step through all test steps
2. [WW17.1: Done] User able to select test case to run
3. [WW17.2: Done] User able to select a range of test case to run
4. [Done] Test result prompt should have status: Passed, Failed, Skipped, Blocked.
5. [WW17.2: WIP] Implement a check mechanism to check tests that has been executed
    a. If executed (with results), skip
6. [Done] Script able to populate all test results into a new json file.
    a. If test_results.json exist, script will prompt warning and exit.
    b. Added "-o/--output" option to allow user to save test result into new file name.

Work archived in GitHub:
https://github.com/rebeccasf/openembedded-core/blob/devel/scripts/manual-test-helper

@Rebecca the mentioned github link is not working. It gives page not found error (404). I need more information to understand and analyse it this bug.

In bugzilla, information is not available about the test cases/test plan and json file.

(In reply to comment #6)
> @Rebecca the mentioned github link is not working. It gives page not found
> error (404). I need more information to understand and analyse it this bug.
> 
> In bugzilla, information is not available about the test cases/test plan and
> json file.

The link is outdated and not longer being used. Provide you with latest link for your references.

Work archived in poky-contrib:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/scripts/manual-test-helper?h=mazliana/qa-revamp-manual

Please find below information for running the manual-test-helper script.

User manual:
1. To run as normal process
$ ./manual-test-helper -F <test suite json file location> 
<test suite json file location>: Currently tested with ../meta/lib/oeqa/manual/bitbake/bitbake_testcases.json
 
2. To print out all test cases in JSON file
$ ./manual-test-helper -F <test suite json file location> -S
$ ./manual-test-helper -F <test suite json file location> --scan
 
3. To run specific test case (-tc or --testcase)
$ ./manual-test-helper -F <test suite json file location> -tc <test case name>
<test case name>: Use the output from “--scan”
 
4. To run a range of test cases, say: we want to run test case #3 to #6 (-tr or --testrange)
$ ./manual-test-helper -F <test suite json file location> -tr 3 6
 
5. To route test result to another file, use “-o/--output” option
$ ./manual-test-helper -F <test suite json file location> -o “<file name>”
 
6. You are able to rerun specific test case by providing the test result file (-r or --rerun)
$ ./manual-test-helper -F <test suite json file location> -o “<file name>” -r <testcasename>
 
7. If you have a partial complete test result, you can resume test (-R or --resume)
$ ./manual-test-helper -F <test suite json file location> -o “<file name>” –R

8. If you want run the test plan, you need to include the test_plan.conf 
 $ ./manual-test-helper -F <test suite json file location> -tp  <test plan conf file location>

9. If you want resume from execute the test plan.
$ ./manual-test-helper -F <test suite json file location> -tp  <test plan conf file location> -R

Prototype implementation Plan [Status]:

1. [Done] The script able to provide user to store failure log.
2. [Done] The script able to delete the log file when update the test cases from failed to passed/skipped/blocked.
3. [Done] Standardize test result format among automated and manual tests.
4. [Done] Completed enable function for test plan- The user need add which test cases in test_plan.conf to exclude from execution.

Enhancement:
1. [WW40.5 WIP] The script required refactoring given the changes to OEQA where OEQA will be able to write testresult to json.

Work archived in GitHub:
https://github.com/mazliana/openembedded-core/blob/manual-test-helper/scripts/manual-test-helper

Recently, roadmap was changes where we plan to upstream manual-test-helper with bare-minimum function. Manual-test-helper is one of an alternative test case management tool of Testopia by naming as "manualexecution". 

Bare-minimum function refer to function where the user can only execute all of the test cases in baseline command, which consists of user guideline steps and the expected results. The last step will ask user to provide their input to execute result. The input options are passed/failed/skipped status. The result given will be written in testresults.json including log error from the user input and configuration if there is any. The output test result for json file is created by using OEQA library.
 
The configuration part is manually key-in by the user. The system allow user to specify how many configuration they want to add and they need to define the required configuration name and value pair. In QA perspective, "configuration" means the test environments  and parameters used during QA setup before testing can be carry out. Example of configurations: image used for boot up, host machine distro used, poky configurations, etc. 
 
The purpose of adding the configuration is to standardize the output test result format between automation and manual execution. 
 
The functionality of manual-test-helper (bare-minimum function):
1. Execute all test cases
2. Create log file if result failed
3. The users input result only accept passed/failed/skipped. When users put other than passed/failed, it will detect as skipped.
	Ex: When users press enter or number, the input detect as skipped.
4. Reuse OEQA library to create test_result.json.
5. Users are able to know the total number of test cases that need to be executed.
6. The configuration part is manually key-in by the user. 

To use these scripts, first source oe environment, then run the entry point script to look for help.
        $ test-case-mgmt

To execute manual test cases, execute the below
        $ test-case-mgmt manualexecution <manualjsonfile>

By default testresults.json store in <build_dir>/tmp/log/manual


[1] scripts/test-case-mgmt: add "manualexecution" as a tool : https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/scripts/test-case-mgmt?h=mazliana/test-case-mgmt-store-report-revised
[2] script/lib/testcasemgmt/manualexecution.py : manual helper script with bare-minimum function : https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/scripts/lib/testcasemgmt/manualexecution.py?h=mazliana/test-case-mgmt-store-report-revised

Submitted revised patch to manage manual execution. 
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278488.html