Bug 8170

Summary: testimage does not correctly support testing LSB images such as full-cmdline
Product: [QA/Testing] Runtime Testing Reporter: Saul Wold <sgw>
Component: generalAssignee: Costin <costin.c.constantin>
Status: VERIFIED FIXED QA Contact: Cristina Agurida <cristina-danielax.agurida>
Severity: critical    
Priority: High CC: bluelightning, bogdanx.a.voiculescu, costin.c.constantin, elizabeth.flanagan, richard.purdie, ross.burton
Version: unspecified   
Target Milestone: 1.9 M3   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: New (Never tested)
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on:    
Bug Blocks: 8130    

Description Saul Wold 2015-08-16 20:00:19 UTC
Based on Richards request to setup the Autobuilder for lsb testing via testimage, I attempted a build with DISTRO=poky-lsb and core-image-full-cmdline. When I ran the -c testimage, I got the following failures.

This clearly shows that we need some modifications to the test suite for correctly testing LSB images:


| ======================================================================
| FAIL: test_pam (oeqa.runtime.pam.PamBasicTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/srv/hdd/poky/meta/lib/oeqa/utils/decorators.py", line 102, in wrapped_f
|     return func(*args)
|   File "/srv/hdd/poky/meta/lib/oeqa/utils/decorators.py", line 90, in wrapped_f
|     return f(*args)
|   File "/srv/hdd/poky/meta/lib/oeqa/runtime/pam.py", line 25, in test_pam
|     self.assertEqual(status, 0, msg = "useradd command does not work as expected. Status and output:%s and %s" %(status, output))
| AssertionError: 127 != 0 : useradd command does not work as expected. Status and output:127 and sh: useradd: command not found
| 
| ======================================================================
| FAIL: test_skeleton_availability (oeqa.runtime.skeletoninit.SkeletonBasicTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/srv/hdd/poky/meta/lib/oeqa/utils/decorators.py", line 90, in wrapped_f
|     return f(*args)
|   File "/srv/hdd/poky/meta/lib/oeqa/runtime/skeletoninit.py", line 19, in test_skeleton_availability
|     self.assertEqual(status, 0, msg = "skeleton init script not found. Output:\n%s " % output)
| AssertionError: 2 != 0 : skeleton init script not found. Output:
| ls: cannot access /etc/init.d/skeleton: No such file or directory
| 
| ======================================================================
| FAIL: test_skeleton_script (oeqa.runtime.skeletoninit.SkeletonBasicTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/srv/hdd/poky/meta/lib/oeqa/utils/decorators.py", line 102, in wrapped_f
|     return func(*args)
|   File "/srv/hdd/poky/meta/lib/oeqa/utils/decorators.py", line 90, in wrapped_f
|     return f(*args)
|   File "/srv/hdd/poky/meta/lib/oeqa/runtime/skeletoninit.py", line 29, in test_skeleton_script
|     self.assertEqual(status, 0, msg = "Skeleton script could not be started:\n%s\n%s" % (output1, output2))
| AssertionError: 1 != 0 : Skeleton script could not be started:
| sh: /etc/init.d/skeleton: No such file or directory
| 
| 
| ----------------------------------------------------------------------
| Ran 21 tests in 36.737s
| 
This will block the proper implementation of this on the Autobuilder as all LSB images will be red, therefore I am marking this as a Critical High for M3.
Comment 1 Paul Eggleton 2015-08-17 09:47:08 UTC
A minor note - core-image-full-cmdline is not an LSB image. It is used as the basis for LSB images though.
Comment 2 Saul Wold 2015-08-17 18:31:47 UTC
It seems that the oeRuntimeTest.hasPackage("service") matches a package group: 
packagegroup-core-full-cmdline-sys-services, not the meta-skeleton service 
package, so part of the problem might be that hasPackage() needs to exact 
matching or use some regular expression format.
Comment 3 Paul Eggleton 2015-08-17 18:36:52 UTC
It does look like it's doing a re.search() on the entire package list file contents - which seems wrong to me especially given how it is being used in the various tests. It ought to be doing a re.match() on each item individually if we even need regex support.
Comment 4 Costin 2015-08-20 07:11:54 UTC
i made a test by replacing re.search() with re.match(). This time testimage signals skipped packages and tests go on as expected. 
Still i spotted an issue with useadd but i guess this is not related to this bug:

 ======================================================================
| FAIL: test_pam (oeqa.runtime.pam.PamBasicTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/media/yp_builds/5poky/meta/lib/oeqa/utils/decorators.py", line 102, in wrapped_f
|     return func(*args)
|   File "/media/yp_builds/5poky/meta/lib/oeqa/utils/decorators.py", line 90, in wrapped_f
|     return f(*args)
|   File "/media/yp_builds/5poky/meta/lib/oeqa/runtime/pam.py", line 25, in test_pam
|     self.assertEqual(status, 0, msg = "useradd command does not work as expected. Status and output:%s and %s" %(status, output))
| AssertionError: 127 != 0 : useradd command does not work as expected. Status and output:127 and sh: useradd: command not found
|
| ----------------------------------------------------------------------
Comment 5 Costin 2015-08-20 07:33:10 UTC
related to useradd the resulting image contains luseradd so this is why the related assert to the useradd --help fails
Comment 6 Paul Eggleton 2015-08-20 08:01:22 UTC
It needs to be more than just a swap of re.search() with re.match() - we also need to do the matching on individual items rather than the whole list.
Comment 7 Costin 2015-08-20 14:17:31 UTC
Hi Paul,

I followed your recommendation and did a split by each line.
However i got a situation similar to the initial one for syslog. I built the same image as on comment#1 and in  the .manifest file i can find busybox-syslog. So even if the service exists, testimage is not treating it: "skipped 'syslog: No syslog package in image'"
Should i add an exception for busybox-* ?

Thanks
Comment 8 Paul Eggleton 2015-08-21 08:23:03 UTC
We don't want a hardcoded exception here - if the test genuinely works for both syslog and busybox-syslog then you will have to check *in the test* for both packages.
Comment 9 Costin 2015-08-21 12:23:12 UTC
patch submitted upstream
Comment 11 Cristina Agurida 2015-09-03 11:52:08 UTC
Verified on master: 71b0568fa43285f0946fae93fb43cea5f3bbecec