Bug 10964 - testimage: Test suite is not running when images are downloaded from Autobuilder
Summary: testimage: Test suite is not running when images are downloaded from Autobuilder
Status: VERIFIED FIXED
Alias: None
Product: BSPs
Classification: Build System, Metadata & Runtime
Component: bsps-oe-core (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: High normal
Target Milestone: 2.3 M2
Assignee: Mariano Lopez
QA Contact: Jose Perez C
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-01-25 15:06 UTC by Jose Perez C
Modified: 2017-02-08 20:25 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jose Perez C 2017-01-25 15:06:58 UTC 
Commit: 62d7d4130202d8ede16abf9e7d779361ca70847e
AB images: http://autobuilder.yoctoproject.org/pub/releases/yocto-2.3_M2.rc1/machines/

STEPS TO REPRODUCE: 
  1. Download image to test form AB 
  2. Prepare the environment according to the image tested 
     $ bitbake rpm psplash
     $ bitbake package-index
  3. $ bitbake core-image-sato-sdk -c testimage


EXPECTED RESULT:
 - Test should be executed correctly 

ACTUAL RESULT: 
 - Test are not being executed
Comment 1 Mariano Lopez 2017-01-25 17:16:56 UTC 
This happens because there are some hardcoded paths in rpm and smart tests. In order to work around this, it is necessary to modify <image>.tesdata.json.
You need to change the WORKDIR and DEPLOY_DIR to fit your current environment (bitbake -e can help you to get your current environment).

Also I found two bugs within smart and connman tests. You can find the fixes in the following commits in poky-contrib:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/bug10964&id=c975cbf858065c4b89fb25e81081b5ecd7abc632

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/bug10964&id=acd70b408f334295436908b2a9fa37a66f71cc75
Comment 2 Jose Perez C 2017-01-25 22:43:37 UTC 
With last 3 patches from http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/bug10964 test are executed correctly but noticed that there were some missing test cases executed for the images tested as follows :

core-image-sato-sdk:

201	syslogd workable (auto)
202	syslog configurable (auto)
205	cvs project compile in sdk image (auto)
206	iptables project compile in sdk image (auto)
207	sudoku-savant project compile in sdk image (auto)
1149	test syslog logger (auto)
1150	test syslog restart (auto)
Comment 3 Mariano Lopez 2017-01-26 21:31:03 UTC 
(In reply to comment #2)
> With last 3 patches from
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mariano/
> bug10964 test are executed correctly but noticed that there were some
> missing test cases executed for the images tested as follows :
> 
> core-image-sato-sdk:
> 
> 201	syslogd workable (auto)
> 202	syslog configurable (auto)
> 205	cvs project compile in sdk image (auto)
> 206	iptables project compile in sdk image (auto)
> 207	sudoku-savant project compile in sdk image (auto)
> 1149	test syslog logger (auto)
> 1150	test syslog restart (auto)

DEFAULT_TEST_SUITES were missing some tests for core-image-sato-sdk and core-image-lsb-sdk.

For syslog it seems the test is not loaded on debian/ubuntu machines.
Comment 4 Mariano Lopez 2017-01-27 20:20:23 UTC 
Sent v2 patches: https://patchwork.openembedded.org/series/4986/
Comment 5 Mariano Lopez 2017-01-30 16:18:05 UTC 
Sent v3. I hope patchtest won't complain this time.

https://patchwork.openembedded.org/series/4986/
Comment 7 Jose Perez C 2017-02-08 20:25:09 UTC 
Verified on 2.3 M2 rc3 is working correctly 

commit: 226a508da955439b881b2f0a544a3aee76e59919