7833 2015-06-01 13:38:00 +0000 tc.filesdir can't handle ./files/ folder under other layers 2015-08-20 04:33:53 +0000 1 1 10 QA/Testing Runtime Testing general 1.8.1 All Multiple VERIFIED FIXED Medium+ enhancement 1.9 M2 1 cathy.shen costin.c.constantin jing.j.wang cathy.shen --- --- Yes (doc changes required) oldest_to_newest 51383 0 cathy.shen 2015-06-01 13:38:00 +0000 ts.filesdir can't find the files under ./files/ folder if the test script is under other layer and trying to find the dependency in the ./files/ folder of the same layer. Currently, we are using workaround: os.path.join(os.path.dirname(__file__), 'files', <tc_dependent_file_name>) 51520 1 cathy.shen 2015-06-05 03:55:38 +0000 JingWang should have a patch to fix this. 51536 2 jing.j.wang 2015-06-07 23:03:44 +0000 Currently, tc.file_dir is only for /lib/oeqa/runtime/files under oe core layer. For file_dir under other layer, A quick solution is TC itself handle them directly, but not use tc.file_dir. 51537 3 cathy.shen 2015-06-08 01:15:24 +0000 Jing, do you mean there is no pacth for this, and are you recomending to use below method in test case? Pls. confirm. os.path.join(os.path.dirname(__file__), 'files', <tc_dependent_file_name>) Constin, as the ticket owner, what's your comments? 51563 4 costin.c.constantin 2015-06-08 21:10:52 +0000 Hello Cathy/ Jing, As the testimage framework is right now, it can accept tests form other layers. Downwards is an example. Before that, i would like to point out that we use meta/lib/oeqa/runtime trying to standardise the way we made tests for our Yocto Project QA purposes. Example : PREREQUISITES: Note, my poky dir is named 3poky 1) create a new layer with a minimalistic layer.conf file (just for testing purposes in my case) cosscat@cosscat:/media/cosscat/Touro/3poky/meta-test/conf$ cat layer.conf #add to BBPATH BBPATH .= ":${LAYERDIR}" 2) create lib/oeqa/runtime dir structure with __init__.py in it relative to meta-test path: cosscat@cosscat:/media/cosscat/Touro/3poky/meta-test/lib/oeqa/runtime$ cat __init__.py # Enable other layers to have tests in the same named directory from pkgutil import extend_path __path__ = extend_path(__path__, __name__) Also add few .py test files here. I added mytest.py and later on 2mytest.py: cosscat@cosscat:/media/cosscat/Touro/3poky/meta-test/lib/oeqa/runtime$ ls __init__.py mytest.py mytest.pyc I added intentionally self.assertEqual(1, 0, msg = "This is just for testing purposes ...") in mytest.py 3) add meta-test to bblayers.conf: cosscat@cosscat:/media/cosscat/Touro/3poky/build$ cat conf/bblayers.conf ... BBLAYERS ?= " \ /media/cosscat/Touro/3poky/meta \ /media/cosscat/Touro/3poky/meta-yocto \ /media/cosscat/Touro/3poky/meta-yocto-bsp \ /media/cosscat/Touro/3poky/meta-test \ " ... 4) inherit testimage.bbclass in local.conf and set TEST_SUITES variable (i will present you my test scenario, using qemux86 as machine): ... MACHINE ??= "qemux86" ... EXTRA_IMAGE_FEATURES = "debug-tweaks ssh-server-openssh" # i used ssh-server-openssh in order to have ssh available in qemu image when building core-image-minimal. # it was faster to build an image-minimal than sato and suffices for our testing purposes ... CONF_VERSION = "1" INHERIT += "testimage" TEST_SUITES = "auto" 5) add support for running commands via sudo without being asked for password (used later on to launch runqemu). This is valid for Ubuntu: cosscat@cosscat:/media/cosscat/Touro/3poky/build$ sudo visudo ... # my group name is cosscat. if you want only for a single user, erase the "%" %cosscat ALL=(ALL) NOPASSWD:ALL TESTING ITSELF: Method 1: using the TEST_SUITES = "auto" in local.conf. When using TEST_SUITES = "auto" in local.conf the testimage framework will use the __init__.py file found in meta*/lib/oeqa/runtime to automatically add all test .py files it finds here. Thus more layers can be used, the condition being for them to contain a lib/oeqa/runtime dir relative to the meta-name_of_layer. In order for "auto" keyword to work with other layers, i had to do a small change to testimage.bbclass. I will submit a patch for it and let you have it soon. run the tests itself (in my case just one, from mytest.py): cosscat@cosscat:/media/cosscat/Touro/3poky/build$ bitbake core-image-minimal -c testimage ... NOTE: Test modules ['oeqa.runtime.mytest'] | NOTE: Found 1 tests | test_scp_file (oeqa.runtime.mytest.ScpTest) ... FAIL # the name is test_scp_file because mytest.py is a copy of scp.py, with small mods to make it fail | | ====================================================================== | FAIL: test_scp_file (oeqa.runtime.mytest.ScpTest) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "/media/cosscat/Touro/3poky/meta/lib/oeqa/utils/decorators.py", line 100, in wrapped_f | return func(*args) | File "/media/cosscat/Touro/3poky/meta/lib/oeqa/utils/decorators.py", line 88, in wrapped_f | return f(*args) | File "/media/cosscat/Touro/3poky/meta-test/lib/oeqa/runtime/mytest.py", line 20, in test_scp_file | self.assertEqual(1, 0, msg = "This is just for testing purposes ...") | AssertionError: 1 != 0 : This is just for testing purposes ... | | ---------------------------------------------------------------------- ... Method 2: using a list of test names (that correspond to test_names.py from lib/oeqa/runtime): - you can put the TEST_SUITES variable in local.conf or in a separate file e. test_run.txt: TEST_SUITES ?= "mytest 2mytest" # created a second file 2mytest bitbake core-image-minimal -c testimage # when setting in local.conf or bitbake core-image-minimal -c testimage -r test_run.txt NOTE: Found 2 tests | test_scp_file (oeqa.runtime.mytest.ScpTest) ... FAIL | test_scp_file (oeqa.runtime.2mytest.ScpTest) ... FAIL | | ====================================================================== | FAIL: test_scp_file (oeqa.runtime.mytest.ScpTest) | ---------------------------------------------------------------------- |... | self.assertEqual(1, 0, msg = "This is just for testing purposes ...") | AssertionError: 1 != 0 : This is just for testing purposes ... | | ====================================================================== | FAIL: test_scp_file (oeqa.runtime.2mytest.ScpTest) | ---------------------------------------------------------------------- ... | AssertionError: 1 != 0 : This is from inside 2mytest.py file | | ---------------------------------------------------------------------- | Ran 2 tests in 4.017s | Please let me know if there is something you would like to know related to adding tests into an other layer. Costin 51564 5 costin.c.constantin 2015-06-08 21:14:36 +0000 Hi Cathy/ Jing, I forgot to add to my previous comment that i renamed for TEST_SUITES = "auto" the meta/lib/oeqa/__init__.py to something else, preventing the framework to consider tests from this path also. Costin 51770 6 costin.c.constantin 2015-06-17 20:06:30 +0000 Hi Jing, Cathy, After trying several possibilities, i came to the conclusion that the way to have files dir available in other meta* layer is by setting the tc.filesdir in the setUpModule() method of each .py test suite file. In my case i have a poky/meta-teste/lib/oeqa/runtime/files dir and the oeqa.runtime.test.py looks like: ... def setUpModule(): oeRuntimeTest.tc.filesdir = os.path.join(os.path.dirname(os.path.abspath(__file__)),"files") ... The approach is similar to comment#3, the difference being we still use oeRuntimeTest.tc.filesdir Thank you, Costin 52158 7 jing.j.wang 2015-07-07 06:46:24 +0000 This solution looks fine for me 52951 8 costin.c.constantin 2015-08-10 14:57:16 +0000 marking as resolved, considering last comment 52952 9 costin.c.constantin 2015-08-10 14:57:45 +0000 marking as verified