Bug 11737

Summary: include selftest tests for devtool import/export plugins
Product: [QA/Testing] Build Testing Reporter: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Component: generalAssignee: Aaron Chan <aaron.chun.yew.chan>
Status: ACCEPTED --- QA Contact: Aaron Chan <aaron.chun.yew.chan>
Severity: normal    
Priority: Medium+ CC: aaron.chun.yew.chan, apoorv.sangal, jose.perez.carranza, leonardo.sandoval.gonzalez, randy.macleod, sgw
Version: 2.4   
Target Milestone: 5.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 10510    
Bug Blocks:    

Description Leonardo Sandoval Gonzalez 2017-06-29 16:51:17 UTC 
Once the series including export/import devtool plugins land to master, the system should exercise them. The most basic features should be tested based on [1], which I actually tested but did not fully work when run the whole test case (tear down commands were failing).

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=lsandov1/devtool-import-export-selftest
Comment 1 Jose Perez C 2017-08-18 14:04:33 UTC 
Moved to 2.5 as this feature is not yet integrated
Comment 2 Aaron Chan 2018-03-12 22:41:46 UTC 
With reference to link below:
http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-ubuntu1604/builds/58/steps/Running%20oe-selftest/logs/stdio
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lsandov1/devtool-import-export-selftest&id=4b0e8df2ecb7f6a90add6bf4d2839bf5d8478a4d


2018-03-13 07:18:38,192 - oe-selftest - INFO -   test_devtool_deploy_target (devtool.DevtoolTests) ... OK (149.287s)
2018-03-13 07:19:17,697 - oe-selftest - INFO -   test_devtool_export (devtool.DevtoolTests) ... OK (39.503s)
2018-03-13 07:19:56,857 - oe-selftest - WARNING - tearDown commands have failed: rm workspace-export*.tar.gz
2018-03-13 07:19:56,929 - oe-selftest - INFO -   test_devtool_export_exclude_all (devtool.DevtoolTests) ... OK (39.231s)
2018-03-13 07:20:34,443 - oe-selftest - INFO -   test_devtool_export_exclude_specific_file (devtool.DevtoolTests) ... OK (37.513s)
2018-03-13 07:21:11,969 - oe-selftest - INFO -   test_devtool_export_include (devtool.DevtoolTests) ... OK (37.523s)
2018-03-13 07:21:51,570 - oe-selftest - INFO -   test_devtool_export_overwrite_file (devtool.DevtoolTests) ... OK (39.598s)
2018-03-13 07:22:09,454 - oe-selftest - INFO -   test_devtool_extract (devtool.DevtoolTests) ... OK (17.881s)
2018-03-13 07:22:25,818 - oe-selftest - INFO -   test_devtool_extract_virtual (devtool.DevtoolTests) ... OK (16.364s)
2018-03-13 07:23:06,201 - oe-selftest - INFO -   test_devtool_finish_modify_origlayer (devtool.DevtoolTests) ... OK (40.382s)
2018-03-13 07:23:45,617 - oe-selftest - INFO -   test_devtool_finish_modify_otherlayer (devtool.DevtoolTests) ... OK (39.415s)
2018-03-13 07:24:22,425 - oe-selftest - INFO -   test_devtool_finish_upgrade_origlayer (devtool.DevtoolTests) ... OK (36.807s)
2018-03-13 07:25:02,547 - oe-selftest - INFO -   test_devtool_finish_upgrade_otherlayer (devtool.DevtoolTests) ... OK (40.121s)
2018-03-13 07:25:55,764 - oe-selftest - WARNING - tearDown commands have failed: bitbake-layers remove-layer *//tmp/import7syded0e
2018-03-13 07:25:55,898 - oe-selftest - INFO -   test_devtool_import (devtool.DevtoolTests) ... OK (53.350s)

Here are list of testcases below run for devtool.DevtoolTests are NOT included with Testcase ID and yet to be merged into the HEAD on master branch.
All the testcase mentioned below are tested in local PG autobuilder and found to be passing. All other testcases are failing using this branch lsandov1/devtool-import-export-selftest.

a) test_devtool_export
b) test_devtool_export_include
c) test_devtool_export_exclude_all
d) test_devtool_export_exclude_specific_file
e) test_devtool_export_overwrite_file
f) test_devtool_import

2018-03-13 07:27:13,844 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_create_workspace - Testcase 1158: PASSED
2018-03-13 07:27:13,844 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_add - Testcase 1159: PASSED
2018-03-13 07:27:13,845 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_add_fetch - Testcase 1160: PASSED
2018-03-13 07:27:13,846 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_add_fetch_git - Testcase 1161: PASSED
2018-03-13 07:27:13,847 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_add_fetch_simple - Testcase 1391: PASSED
2018-03-13 07:27:13,848 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_add_git_local - Testcase 1423: PASSED
2018-03-13 07:27:13,849 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_add_library - Testcase 1162: PASSED
2018-03-13 07:27:13,850 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_build_image - Testcase 1366: PASSED
2018-03-13 07:27:13,851 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_buildclean - Testcase 1620: PASSED
2018-03-13 07:27:13,852 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_deploy_target - Testcase 1272: PASSED
2018-03-13 07:27:13,853 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_export - Testcase -1: PASSED
2018-03-13 07:27:13,853 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_export_exclude_all - Testcase -1: PASSED
2018-03-13 07:27:13,854 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_export_exclude_specific_file - Testcase -1: PASSED
2018-03-13 07:27:13,855 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_export_include - Testcase -1: PASSED
2018-03-13 07:27:13,856 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_export_overwrite_file - Testcase -1: PASSED
2018-03-13 07:27:13,857 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_extract - Testcase 1163: PASSED
2018-03-13 07:27:13,858 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_extract_virtual - Testcase 1379: PASSED
2018-03-13 07:27:13,859 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_finish_modify_origlayer - Testcase 1621: PASSED
2018-03-13 07:27:13,860 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_finish_modify_otherlayer - Testcase 1622: PASSED
2018-03-13 07:27:13,861 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_finish_upgrade_origlayer - Testcase 1623: PASSED
2018-03-13 07:27:13,861 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_finish_upgrade_otherlayer - Testcase 1624: PASSED
2018-03-13 07:27:13,862 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_import - Testcase -1: PASSED

Since these additional testcases are passing in lsandov1/devtool-import-export-selftest, it should be merged into master branch to be included into 2.5 M3 and beyond.
Comment 3 Jose Perez C 2018-03-13 06:26:10 UTC 
(In reply to comment #2)
> With reference to link below:
> http://centaurus01.png.intel.com:8010/builders/nightly-oe-selftest-
> ubuntu1604/builds/58/steps/Running%20oe-selftest/logs/stdio
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lsandov1/
> devtool-import-export-selftest&id=4b0e8df2ecb7f6a90add6bf4d2839bf5d8478a4d
> 
> 
> 2018-03-13 07:18:38,192 - oe-selftest - INFO -   test_devtool_deploy_target
> (devtool.DevtoolTests) ... OK (149.287s)
> 2018-03-13 07:19:17,697 - oe-selftest - INFO -   test_devtool_export
> (devtool.DevtoolTests) ... OK (39.503s)
> 2018-03-13 07:19:56,857 - oe-selftest - WARNING - tearDown commands have
> failed: rm workspace-export*.tar.gz
> 2018-03-13 07:19:56,929 - oe-selftest - INFO -  
> test_devtool_export_exclude_all (devtool.DevtoolTests) ... OK (39.231s)
> 2018-03-13 07:20:34,443 - oe-selftest - INFO -  
> test_devtool_export_exclude_specific_file (devtool.DevtoolTests) ... OK
> (37.513s)
> 2018-03-13 07:21:11,969 - oe-selftest - INFO -   test_devtool_export_include
> (devtool.DevtoolTests) ... OK (37.523s)
> 2018-03-13 07:21:51,570 - oe-selftest - INFO -  
> test_devtool_export_overwrite_file (devtool.DevtoolTests) ... OK (39.598s)
> 2018-03-13 07:22:09,454 - oe-selftest - INFO -   test_devtool_extract
> (devtool.DevtoolTests) ... OK (17.881s)
> 2018-03-13 07:22:25,818 - oe-selftest - INFO -  
> test_devtool_extract_virtual (devtool.DevtoolTests) ... OK (16.364s)
> 2018-03-13 07:23:06,201 - oe-selftest - INFO -  
> test_devtool_finish_modify_origlayer (devtool.DevtoolTests) ... OK (40.382s)
> 2018-03-13 07:23:45,617 - oe-selftest - INFO -  
> test_devtool_finish_modify_otherlayer (devtool.DevtoolTests) ... OK (39.415s)
> 2018-03-13 07:24:22,425 - oe-selftest - INFO -  
> test_devtool_finish_upgrade_origlayer (devtool.DevtoolTests) ... OK (36.807s)
> 2018-03-13 07:25:02,547 - oe-selftest - INFO -  
> test_devtool_finish_upgrade_otherlayer (devtool.DevtoolTests) ... OK
> (40.121s)
> 2018-03-13 07:25:55,764 - oe-selftest - WARNING - tearDown commands have
> failed: bitbake-layers remove-layer *//tmp/import7syded0e
Here is a tearDown command fialing, I think this should be corrected.

> 2018-03-13 07:25:55,898 - oe-selftest - INFO -   test_devtool_import
> (devtool.DevtoolTests) ... OK (53.350s)
> 
> Here are list of testcases below run for devtool.DevtoolTests are NOT
> included with Testcase ID and yet to be merged into the HEAD on master
> branch.
> All the testcase mentioned below are tested in local PG autobuilder and
> found to be passing. All other testcases are failing using this branch
> lsandov1/devtool-import-export-selftest.
Did you rebase this branch on top master? Because this branch is kind of old. 

> 
> a) test_devtool_export
> b) test_devtool_export_include
> c) test_devtool_export_exclude_all
> d) test_devtool_export_exclude_specific_file
> e) test_devtool_export_overwrite_file
> f) test_devtool_import
> 
> 2018-03-13 07:27:13,844 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_create_workspace - Testcase 1158: PASSED
> 2018-03-13 07:27:13,844 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_add - Testcase 1159: PASSED
> 2018-03-13 07:27:13,845 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_add_fetch - Testcase 1160: PASSED
> 2018-03-13 07:27:13,846 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_add_fetch_git - Testcase 1161: PASSED
> 2018-03-13 07:27:13,847 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_add_fetch_simple - Testcase 1391: PASSED
> 2018-03-13 07:27:13,848 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_add_git_local - Testcase 1423: PASSED
> 2018-03-13 07:27:13,849 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_add_library - Testcase 1162: PASSED
> 2018-03-13 07:27:13,850 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_build_image - Testcase 1366: PASSED
> 2018-03-13 07:27:13,851 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_buildclean - Testcase 1620: PASSED
> 2018-03-13 07:27:13,852 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_deploy_target - Testcase 1272: PASSED
> 2018-03-13 07:27:13,853 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_export - Testcase -1: PASSED
> 2018-03-13 07:27:13,853 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_export_exclude_all - Testcase -1: PASSED
> 2018-03-13 07:27:13,854 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_export_exclude_specific_file - Testcase
> -1: PASSED
> 2018-03-13 07:27:13,855 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_export_include - Testcase -1: PASSED
> 2018-03-13 07:27:13,856 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_export_overwrite_file - Testcase -1: PASSED
> 2018-03-13 07:27:13,857 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_extract - Testcase 1163: PASSED
> 2018-03-13 07:27:13,858 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_extract_virtual - Testcase 1379: PASSED
> 2018-03-13 07:27:13,859 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_finish_modify_origlayer - Testcase 1621:
> PASSED
> 2018-03-13 07:27:13,860 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_finish_modify_otherlayer - Testcase 1622:
> PASSED
> 2018-03-13 07:27:13,861 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_finish_upgrade_origlayer - Testcase 1623:
> PASSED
> 2018-03-13 07:27:13,861 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_finish_upgrade_otherlayer - Testcase 1624:
> PASSED
> 2018-03-13 07:27:13,862 - oe-selftest - INFO - RESULTS -
> devtool.DevtoolTests.test_devtool_import - Testcase -1: PASSED
> 
> Since these additional testcases are passing in
> lsandov1/devtool-import-export-selftest, it should be merged into master
> branch to be included into 2.5 M3 and beyond.

I think the test cases that are failing should be reviewed as they may be failing due the new test test cases added.
Comment 4 Randy MacLeod 2023-10-24 14:24:22 UTC 
Bulk move from 4.99 or 0.00 to 5.99
Comment 5 Randy MacLeod 2025-02-14 16:46:26 UTC 
bulk change to add Saul's non-WR address.
Comment 6 Randy MacLeod 2025-02-14 16:48:25 UTC 
Bulk change: Remove Saul's old WR address.