Bug 7833

JingWang should have a patch to fix this.

Currently, tc.file_dir is only for /lib/oeqa/runtime/files under oe core layer. For file_dir under other layer, A quick solution is TC itself handle them directly, but not use tc.file_dir.

Jing, do you mean there is no pacth for this, and are you recomending to use below method in test case? Pls. confirm.
     os.path.join(os.path.dirname(__file__), 'files', <tc_dependent_file_name>)

Constin, as the ticket owner, what's your comments?

Hello Cathy/ Jing,

As the testimage framework is right now, it can accept tests form other layers. Downwards is an example. Before that, i would like to point out that we use meta/lib/oeqa/runtime trying to standardise the way we made tests for our Yocto Project QA purposes.

Example :

PREREQUISITES:
Note, my poky dir is named 3poky 

1) create a new layer with a minimalistic layer.conf file (just for testing purposes in my case)
cosscat@cosscat:/media/cosscat/Touro/3poky/meta-test/conf$ cat layer.conf 
#add to BBPATH
BBPATH .= ":${LAYERDIR}"

2) create lib/oeqa/runtime dir structure with __init__.py in it relative to meta-test path:
cosscat@cosscat:/media/cosscat/Touro/3poky/meta-test/lib/oeqa/runtime$ cat __init__.py 
# Enable other layers to have tests in the same named directory
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Also add few .py test files here. I added mytest.py and later on 2mytest.py:
cosscat@cosscat:/media/cosscat/Touro/3poky/meta-test/lib/oeqa/runtime$ ls
__init__.py  mytest.py  mytest.pyc

I added intentionally self.assertEqual(1, 0, msg = "This is just for testing purposes ...") in mytest.py 

3) add meta-test to bblayers.conf:
cosscat@cosscat:/media/cosscat/Touro/3poky/build$ cat conf/bblayers.conf 
...
BBLAYERS ?= " \
  /media/cosscat/Touro/3poky/meta \
  /media/cosscat/Touro/3poky/meta-yocto \
  /media/cosscat/Touro/3poky/meta-yocto-bsp \
  /media/cosscat/Touro/3poky/meta-test \
  "
...

4) inherit testimage.bbclass in local.conf and set TEST_SUITES variable (i will present you my test scenario, using qemux86 as machine):
...
MACHINE ??= "qemux86"
...
EXTRA_IMAGE_FEATURES = "debug-tweaks ssh-server-openssh"

# i used ssh-server-openssh in order to have ssh available in qemu image when building core-image-minimal. 
# it was faster to build an image-minimal than sato and suffices  for our testing purposes

...
CONF_VERSION = "1"
INHERIT += "testimage"
TEST_SUITES = "auto"


5) add support for running commands via sudo without being asked for password (used later on to launch runqemu). This is valid for Ubuntu:
cosscat@cosscat:/media/cosscat/Touro/3poky/build$ sudo visudo
...
# my group name is cosscat. if you want only for a single user, erase the "%"
%cosscat ALL=(ALL) NOPASSWD:ALL


TESTING ITSELF:

Method 1: using the TEST_SUITES = "auto" in local.conf.  

When using TEST_SUITES = "auto" in local.conf the testimage framework will use the __init__.py file found in meta*/lib/oeqa/runtime to automatically add all test .py files it finds here. Thus more layers can be used, the condition being for them to contain a lib/oeqa/runtime dir relative to the meta-name_of_layer. In order for "auto" keyword to work with other layers, i had to do a small change to testimage.bbclass. I will submit a patch for it and let you have it soon.

run the tests itself (in my case just one, from mytest.py):
cosscat@cosscat:/media/cosscat/Touro/3poky/build$ bitbake core-image-minimal -c testimage
...
NOTE: Test modules  ['oeqa.runtime.mytest']
| NOTE: Found 1 tests
| test_scp_file (oeqa.runtime.mytest.ScpTest) ... FAIL # the name is test_scp_file because mytest.py is a copy of scp.py, with small mods to make it fail
| 
| ======================================================================
| FAIL: test_scp_file (oeqa.runtime.mytest.ScpTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/media/cosscat/Touro/3poky/meta/lib/oeqa/utils/decorators.py", line 100, in wrapped_f
|     return func(*args)
|   File "/media/cosscat/Touro/3poky/meta/lib/oeqa/utils/decorators.py", line 88, in wrapped_f
|     return f(*args)
|   File "/media/cosscat/Touro/3poky/meta-test/lib/oeqa/runtime/mytest.py", line 20, in test_scp_file
|     self.assertEqual(1, 0, msg = "This is just for testing purposes ...")
| AssertionError: 1 != 0 : This is just for testing purposes ...
| 
| ----------------------------------------------------------------------
...
 

Method 2: using a list of test names (that correspond to test_names.py from lib/oeqa/runtime):

- you can put the TEST_SUITES variable in local.conf or in a separate file e. test_run.txt:

TEST_SUITES ?= "mytest 2mytest" # created a second file 2mytest 

bitbake core-image-minimal -c testimage   # when setting in local.conf
or
bitbake core-image-minimal -c testimage -r test_run.txt

NOTE: Found 2 tests
| test_scp_file (oeqa.runtime.mytest.ScpTest) ... FAIL
| test_scp_file (oeqa.runtime.2mytest.ScpTest) ... FAIL
| 
| ======================================================================
| FAIL: test_scp_file (oeqa.runtime.mytest.ScpTest)
| ----------------------------------------------------------------------
|...
|     self.assertEqual(1, 0, msg = "This is just for testing purposes ...")
| AssertionError: 1 != 0 : This is just for testing purposes ...
| 
| ======================================================================
| FAIL: test_scp_file (oeqa.runtime.2mytest.ScpTest)
| ----------------------------------------------------------------------
...
| AssertionError: 1 != 0 :  This is from inside 2mytest.py file
| 
| ----------------------------------------------------------------------
| Ran 2 tests in 4.017s
| 

Please let me know if there is something you would like to know related to adding tests into an other layer.

Costin

Hi Cathy/ Jing,

I forgot to add to my previous comment that i renamed for TEST_SUITES = "auto" the meta/lib/oeqa/__init__.py to something else, preventing the framework to consider tests from this path also. 

Costin

Hi Jing, Cathy,

After trying several possibilities, i came to the conclusion that the way to have files dir available in other meta* layer is by setting the tc.filesdir in the setUpModule() method of each .py test suite file. 
In my case i have a poky/meta-teste/lib/oeqa/runtime/files dir and the oeqa.runtime.test.py looks like:
...
def setUpModule():
    oeRuntimeTest.tc.filesdir = os.path.join(os.path.dirname(os.path.abspath(__file__)),"files")
...

The approach is similar to comment#3, the difference being we still use  oeRuntimeTest.tc.filesdir

Thank you,

Costin

This solution looks fine for me

marking as resolved, considering last comment

marking as verified