Bug 10964

Summary: testimage: Test suite is not running when images are downloaded from Autobuilder
Product: [Build System, Metadata & Runtime] BSPs Reporter: Jose Perez C <jose.perez.carranza>
Component: bsps-oe-coreAssignee: Mariano Lopez <mariano.lopez>
Status: VERIFIED FIXED QA Contact: Jose Perez C <jose.perez.carranza>
Severity: normal    
Priority: High CC: leonardo.sandoval.gonzalez
Version: 2.3   
Target Milestone: 2.3 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Jose Perez C 2017-01-25 15:06:58 UTC 
Commit: 62d7d4130202d8ede16abf9e7d779361ca70847e
AB images: http://autobuilder.yoctoproject.org/pub/releases/yocto-2.3_M2.rc1/machines/

STEPS TO REPRODUCE: 
  1. Download image to test form AB 
  2. Prepare the environment according to the image tested 
     $ bitbake rpm psplash
     $ bitbake package-index
  3. $ bitbake core-image-sato-sdk -c testimage


EXPECTED RESULT:
 - Test should be executed correctly 

ACTUAL RESULT: 
 - Test are not being executed
Comment 1 Mariano Lopez 2017-01-25 17:16:56 UTC 
This happens because there are some hardcoded paths in rpm and smart tests. In order to work around this, it is necessary to modify <image>.tesdata.json.
You need to change the WORKDIR and DEPLOY_DIR to fit your current environment (bitbake -e can help you to get your current environment).

Also I found two bugs within smart and connman tests. You can find the fixes in the following commits in poky-contrib:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/bug10964&id=c975cbf858065c4b89fb25e81081b5ecd7abc632

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/bug10964&id=acd70b408f334295436908b2a9fa37a66f71cc75
Comment 2 Jose Perez C 2017-01-25 22:43:37 UTC 
With last 3 patches from http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/bug10964 test are executed correctly but noticed that there were some missing test cases executed for the images tested as follows :

core-image-sato-sdk:

201	syslogd workable (auto)
202	syslog configurable (auto)
205	cvs project compile in sdk image (auto)
206	iptables project compile in sdk image (auto)
207	sudoku-savant project compile in sdk image (auto)
1149	test syslog logger (auto)
1150	test syslog restart (auto)
Comment 3 Mariano Lopez 2017-01-26 21:31:03 UTC 
(In reply to comment #2)
> With last 3 patches from
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/
> bug10964 test are executed correctly but noticed that there were some
> missing test cases executed for the images tested as follows :
> 
> core-image-sato-sdk:
> 
> 201	syslogd workable (auto)
> 202	syslog configurable (auto)
> 205	cvs project compile in sdk image (auto)
> 206	iptables project compile in sdk image (auto)
> 207	sudoku-savant project compile in sdk image (auto)
> 1149	test syslog logger (auto)
> 1150	test syslog restart (auto)

DEFAULT_TEST_SUITES were missing some tests for core-image-sato-sdk and core-image-lsb-sdk.

For syslog it seems the test is not loaded on debian/ubuntu machines.
Comment 4 Mariano Lopez 2017-01-27 20:20:23 UTC 
Sent v2 patches: https://patchwork.openembedded.org/series/4986/
Comment 5 Mariano Lopez 2017-01-30 16:18:05 UTC 
Sent v3. I hope patchtest won't complain this time.

https://patchwork.openembedded.org/series/4986/
Comment 7 Jose Perez C 2017-02-08 20:25:09 UTC 
Verified on 2.3 M2 rc3 is working correctly 

commit: 226a508da955439b881b2f0a544a3aee76e59919