5055 2013-08-27 16:59:38 +0000 Smart and RPM tests are not skipped correctly when no package manager exists. 2013-08-29 15:06:14 +0000 1 1 10 QA/Testing Runtime Testing general 1.5 x86 Multiple RESOLVED INVALID Low normal 1.5 1 sgw sstncr sstncr --- --- No (bug/feature does not impact docs) oldest_to_newest 35951 0 sgw 2013-08-27 16:59:38 +0000 Should this be an error or skipped? | ERROR | ERROR | test_dmesg (oeqa.runtime.dmesg.DmesgTest) ... ok | | ====================================================================== | ERROR: setUpModule (oeqa.runtime.rpm) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "/srv/ssd/sgw/poky/meta/lib/oeqa/runtime/rpm.py", line 11, in setUpModule | skipModule("rpm module skipped: target doesn't have rpm as primary package manager") | File "/srv/ssd/sgw/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule | "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason)) | Exception: | Test rpm wants to be skipped. | Reason is: rpm module skipped: target doesn't have rpm as primary package manager | Test was required in TEST_SUITES, so either the condition for skipping is wrong | or the image really doesn't have the requred feature/package when it should. | | ====================================================================== | ERROR: setUpModule (oeqa.runtime.smart) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "/srv/ssd/sgw/poky/meta/lib/oeqa/runtime/smart.py", line 11, in setUpModule | skipModule("Image doesn't have smart installed") | File "/srv/ssd/sgw/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule | "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason)) | Exception: | Test smart wants to be skipped. | Reason is: Image doesn't have smart installed | Test was required in TEST_SUITES, so either the condition for skipping is wrong | or the image really doesn't have the requred feature/package when it should. | | ---------------------------------------------------------------------- | Ran 17 tests in 60.221s | | FAILED (errors=2) 35952 1 sstncr 2013-08-27 17:12:57 +0000 Well, it depends. There was a recent patch that changed the behaviour for skipping tests. (a required test was run regardless of conditions, and failed in execution not error out at load time) So, I need to see your local.conf. In short, if a test is added in TEST_SUITES it will now error out, so if you have a sato image with TEST_SUITES = "ping ssh rpm smart" those are considered required tests and none of them are skipped (as a module, individual classes can be skipped as a dependency). It works that way because a test usually does some kind of a check: e.g packagement-manament is in IMAGE_FEATURES but we also know for sure that a certain image (e.g core-image-sato) _must_ have packagement-management in features so we error out assuming that something went wrong and that's not the right image. Oh, btw, rpm and smart are in the defaults for sato (so if this is an ipk/deb build, then you need to overwrite TEST_SUITES completely). Alternatively you could use TEST_SUITES = "ping ssh auto", that will indeed skip non-applicable tests. 35953 2 sstncr 2013-08-27 17:24:24 +0000 One more thing, here it is how it used to work before http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=2dfc0e1cc72491d24536792d4bc1d5f978634671 core-image-sato, TEST_SUITES = "ping ssh gcc" NOTE: Test modules ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.gcc'] NOTE: Found 5 tests test_ping (oeqa.runtime.ping.PingTest) ... ok test_ssh (oeqa.runtime.ssh.SshTest) ... ok WARNING: Test gcc is required, not skipping test_gcc_compile (oeqa.runtime.gcc.GccCompileTest) ... FAIL test_gpp_compile (oeqa.runtime.gcc.GccCompileTest) ... FAIL test_make (oeqa.runtime.gcc.GccCompileTest) ... FAIL ====================================================================== FAIL: test_gcc_compile (oeqa.runtime.gcc.GccCompileTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/gcc.py", line 20, in test_gcc_compile self.assertEqual(status, 0, msg="gcc compile failed, output: %s" % output) AssertionError: gcc compile failed, output: sh: gcc: command not found ====================================================================== FAIL: test_gpp_compile (oeqa.runtime.gcc.GccCompileTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/gcc.py", line 26, in test_gpp_compile self.assertEqual(status, 0, msg="g++ compile failed, output: %s" % output) AssertionError: g++ compile failed, output: sh: g++: command not found ====================================================================== FAIL: test_make (oeqa.runtime.gcc.GccCompileTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/gcc.py", line 32, in test_make self.assertEqual(status, 0, msg="running make failed, output %s" % output) AssertionError: running make failed, output sh: make: command not found ---------------------------------------------------------------------- Ran 5 tests in 11.167s FAILED (failures=3) -- This is how it works now: -- NOTE: Test modules ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.gcc'] NOTE: Found 5 tests test_ping (oeqa.runtime.ping.PingTest) ... ok test_ssh (oeqa.runtime.ssh.SshTest) ... ok ERROR ====================================================================== ERROR: setUpModule (oeqa.runtime.gcc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/stefans/z/poky/meta/lib/oeqa/runtime/gcc.py", line 8, in setUpModule skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES") File "/home/stefans/z/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason)) Exception: Test gcc wants to be skipped. Reason is: Image doesn't have tools-sdk in IMAGE_FEATURES Test was required in TEST_SUITES, so either the condition for skipping is wrong or the image really doesn't have the requred feature/package when it should. ---------------------------------------------------------------------- Ran 2 tests in 6.255s FAILED (errors=1) -- Regardless of the change, here it how it looks with TEST_SUITES = "ping ssh auto": skipped "gcc: Image doesn't have tools-sdk in IMAGE_FEATURES" The disadvantage of auto is that we don't control the order tests run. So in the particular case of rpm/smart most likely it has to do with the fact that those are in the default suite: - from meta/classes/testimage.bbclass: DEFAULT_TEST_SUITES = "ping auto" DEFAULT_TEST_SUITES_pn-core-image-minimal = "ping" DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "ping ssh df connman syslog xorg scp vnc date perl ldd gcc rpm smart dmesg" TEST_SUITES ?= "${DEFAULT_TEST_SUITES}" 36051 3 sstncr 2013-08-29 15:06:14 +0000 So bottom line: "rpm smart" shouldn't be added in TEST_SUITES for ipk/deb images as they are considered mandatory tests and don't get skipped.