| Summary: | [OE-seltest] Some automated test cases are being missed in the test run. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Build Testing | Reporter: | Javier Zepeda <javierx.zepeda.murguia> | ||||
| Component: | general | Assignee: | yeoh ee peng <ee.peng.yeoh> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | minor | ||||||
| Priority: | Medium | CC: | jose.perez.carranza, libertad.gonzalez.de.la.cruz, richard.purdie | ||||
| Version: | unspecified | ||||||
| Target Milestone: | 2.5 M4 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | Don't know | |||||
| Attachments: |
|
||||||
Taking 278 as an example, (https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=278), there is no longer any such test in the OE metadata: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/meta/lib?id=7b5e524c8ea071ca244982af28a39dbe94f1fe35 Test 1434 was also removed: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/meta/lib?id=b799e4a033a968e1a2c977b91a80e8de357c32b1 Not sure about the others but 1434 was removed in March so its been gone for a while... (In reply to comment #1) > Taking 278 as an example, > (https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=278), there is > no longer any such test in the OE metadata: > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/meta/ > lib?id=7b5e524c8ea071ca244982af28a39dbe94f1fe35 > > Test 1434 was also removed: > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/meta/ > lib?id=b799e4a033a968e1a2c977b91a80e8de357c32b1 > > Not sure about the others but 1434 was removed in March so its been gone for > a while... Yes this bug is for the QA team to analize the testcases currently on testopia but no longer executed on AB and discontinue them from the testruns we clone each cycle. After an analysis of Testcases executed by AB against Testopia template, below are my conclusions: ==== Test Cases that need to be DELETED from Testopia template ===== 278 - sanity_unsafe_script_referencesy (auto) 300 - Enable cleanup of WORKDIR (auto) 563 - Check if the directfb image can be built (auto) 925 - Remove the obsolote images (auto) 1101 - Check if efi/gummiboot images can be buit 1103 - Check that wic command can create efi/gummiboot installation images 1349 - wic - test mkgummidisk 1902 - Distrodata.test_checkpkg ==== Test Cases that need to be ADDED to Testopia template ===== 1528 - systemd_boot.Systemdboot.test_image_efi_file 1540 - runtime_test.Postinst.test_verify_postinst 1545 - runtime_test.Postinst.test_postinst_rootfs_and_boot 1602 - eSDK.oeSDKExtSelfTest.test_install_libraries_headers 1603 - eSDK.oeSDKExtSelfTest.test_image_generation_binary_feeds 1655 - wic.Wic.test_bmap_long 2001 - runqemu.RunqemuTests.test_boot_machine 2002 - runqemu.RunqemuTests.test_boot_machine_ext4 2003 - runqemu.RunqemuTests.test_boot_machine_iso 2004 - runqemu.RunqemuTests.test_boot_recipe_image 2005 - runqemu.RunqemuTests.test_boot_recipe_image_vmdk 2006 - runqemu.RunqemuTests.test_boot_recipe_image_vdi 2007 - runqemu.RunqemuTests.test_boot_deploy 2008 - runqemu.RunqemuTests.test_boot_deploy_hddimg 2009 - runqemu.RunqemuTests.test_boot_machine_slirp 2009 - runqemu.RunqemuTests.test_boot_machine_slirp_qcow2 2010 - runqemu.RunqemuTests.test_boot_qemu_boot 2011 - runqemu.RunqemuTests.test_boot_rootfs ==== Test Cases that need change of ID ==== - wic.Wic.test_qemu currently on code (meta/lib/oeqa/selftest/cases/wic.py) has assigned ID 1422 and the real ID is 1424. this should be updated. ==== Test cases that need a new ID on Tesopia and Code ==== - tinfoil.TinfoilTests.test_variable_history - wic.Wic.test_expand_mbr_image - wic.Wic.test_wic_cp_ext - wic.Wic.test_wic_ls_ext - wic.Wic.test_wic_rm_ext Two more test cases that need to be removed from testopia template 1642 - test_postinst_roofs_and_boot 1643 - test_verify_postinst Updated the oeselftest testrun template in Testopia for 2.5 release https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=8521 Added missing testcases to the Testopia template OEQA: runtime_test.Postinst.test_postinst_rootfs_and_boot (1540) OEQA: eSDK.oeSDKExtSelfTest.test_install_libraries_headers (1602) OEQA: eSDK.oeSDKExtSelfTest.test_image_generation_binary_feeds (1603) OEQA: wic.Wic.test_bmap_long (1655) OEQA: runqemu.RunqemuTests.test_boot_machine (2001) OEQA: runqemu.RunqemuTests.test_boot_machine_ext4 (2002) OEQA: runqemu.RunqemuTests.test_boot_machine_iso (2003) OEQA: runqemu.RunqemuTests.test_boot_recipe_image (2004) OEQA: runqemu.RunqemuTests.test_boot_recipe_image_vmdk (2005) OEQA: runqemu.RunqemuTests.test_boot_recipe_image_vdi (2006) OEQA: runqemu.RunqemuTests.test_boot_deploy (2007) OEQA: runqemu.RunqemuTests.test_boot_deploy_hddimg (2008) OEQA: runqemu.RunqemuTests.test_boot_machine_slirp_qcow2 (2009) OEQA: runqemu.RunqemuTests.test_boot_qemu_boot (2010) OEQA: runqemu.RunqemuTests.test_boot_rootfs (2011) |
Created attachment 4076 [details] output log from oe-selftest OE-seltest: Some test cases are not being presented or run, even though, they are set in the Test RUN and they are passed successfully on AB and the image is built successfully. This is seen in all environments: Ubuntu 17.04 Ubuntu 16.04 Opensuse 42.2 Fedora 26 Debian 9 Debian 8 Centos 7 Fedora 25 Here below, the test cases which are being parsed, based on the final OE-seltest log: WARNING:root:./testopia_update.py: Test run 8334, Case 278 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 300 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 563 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 925 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1101 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1103 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1349 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1424 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1434 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1642 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1643 wasn't updated WARNING:root:./testopia_update.py: Test run 8334, Case 1902 wasn't updated Actual: The automated oe-selftest test run is finishing with 95.8% of the tests. Expected: Automated test must be finished at 100% of the test cases. a log is attached.