Bug 8525

Some more examples:

FAIL: test20_wic_image_type (oeqa.selftest.wic.Wic)
----------------------------------------------------------------------
AssertionError: Command 'bitbake  wic-image-minimal' returned non-zero exit status 1:
ERROR: Nothing PROVIDES 'wic-image-minimal'
ERROR: wic-image-minimal was skipped: 'ssh-server-openssh' in IMAGE_FEATURES (added via EXTRA_IMAGE_FEATURES) is not a valid image feature. Valid features: allow-empty-password dbg-pkgs debug-tweaks dev-pkgs doc doc-pkgs empty-root-password package-management post-install-logging ptest-pkgs read-only-rootfs splash staticdev-pkgs

(EXTRA_IMAGE_FEATURES should be wiped)


FAIL: test_sstate_sametune_samesigs (oeqa.selftest.sstatetests.SStateTests)
----------------------------------------------------------------------
AssertionError: Command 'bitbake  world meta-toolchain -S none' returned non-zero exit status 1:
ERROR: Nothing PROVIDES 'pcsc-lite' (but /home/ross/Yocto/meta-oe/meta-multimedia/recipes-dvb/oscam/oscam_svn.bb DEPENDS on or otherwise requires it)
ERROR: Required build target 'oscam' has no buildable providers.
Missing or unbuildable dependency chain was: ['oscam', 'pcsc-lite']

(pass -k to world builds? sanitise layer list so it's only oe-core+meta-selftest?)


FAIL: test_buildhistory_diff (oeqa.selftest.oescripts.BuildhistoryDiffTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ross/Yocto/poky/meta/lib/oeqa/selftest/oescripts.py", line 51, in test_buildhistory_diff
    self.run_buildhistory_operation(target, target_config="PR = \"r0\"", change_bh_location=False, expect_error=True)
  File "/home/ross/Yocto/poky/meta/lib/oeqa/selftest/buildhistory.py", line 41, in run_buildhistory_operation
AssertionError: 0 != 1 : Error expected for global config '' and target config 'PR = "r0"'

I'd honestly prefer if it adapted to the distro config as much as possible; then at least I can be relatively sure when I run it that it's testing what I think it is.

We should came up with a list of variables that can vary in local.conf and expect all oe-selftest tests to pass regardless of their value.

For other variables, not included in the list above tests can rely on the default ones (create just from sourcing the environment).

Benjamin, at this point, this seems like something more appropriate for your team than for me. Please re-assign to the appropriate person.

Not sure how to handle this but I have place a possible solution that checks if DISTRO=poky, if fails, the script just quits without any execution. Basically, this is more align  to Ross comment (https://bugzilla.yoctoproject.org/show_bug.cgi?id=8525#c0) than Paul's (https://bugzilla.yoctoproject.org/show_bug.cgi?id=8525#c2). Before proceeding, let me know your opinion. 

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 72bf4dd..1d11cee 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -126,6 +126,10 @@ def preflight_check():
         log.error("You don't seem to have the meta-selftest layer in BBLAYERS")
         return False
 
+    if get_bb_var("DISTRO") != "poky":
+        log.error("oe-selftest is just intended for Poky distros (DISTRO = \"poky\")")
+        return False
+
     log.info("Running bitbake -p")
     runCmd("bitbake -p")

I'm afraid we cannot do that. oe-selftest needs to be able to run successfully in the default OE distro configuration, which isn't poky. In any case, to make this a little more complicated you could easily have DISTRO = "poky" and then change some aspect of the distro configuration that the tests rely upon from local.conf for example.

(In reply to comment #6)
> I'm afraid we cannot do that. oe-selftest needs to be able to run
> successfully in the default OE distro configuration, which isn't poky. In
> any case, to make this a little more complicated you could easily have
> DISTRO = "poky" and then change some aspect of the distro configuration that
> the tests rely upon from local.conf for example.

Got it. I will try some distros to see the failures we are getting right now, and  make changes based on it.

Created attachment 3431 [details]
oe-selftest log with DISTRO=nodistro

Created attachment 3432 [details]
oe-selftest log with DISTRO=poky-bleeding

Created attachment 3433 [details]
oe-selftest log with DISTRO=poky-lsb

Series sent to two mailing lists because changes affect oe-core and poky

http://lists.openembedded.org/pipermail/openembedded-core/2016-October/127586.html

https://lists.yoctoproject.org/pipermail/poky/2016-October/010759.html

Ross/Joshua have made comments on v1, so sending v2 soon.

V2 series sent to the OE-Core mailing list:

http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129233.html

Series is not yet integrated into any branch and M1 is over, so moving to the next milestone.

Series being reworked following Ross Burton comments. Revision 3 pending to be sent to the mailing list.

V3 series sent to the mailing list [1].

Unfortunately, the proposed series does not cover all tests cases but those test & distro checks that fail have been identified and (bugzilla) tracked.

http://lists.openembedded.org/pipermail/openembedded-core/2017-January/130893.html

Several patches from several revisions have been already merged into master but the following series was sent to the mailing list covering imagefeature and sstatetests. Once this is merged, the bug will be resolved.

http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131632.html

Fix wont make M2 so moving to M3. As mentioned in last comment, the series was just sent, waiting for either comments or merge.

Patches merged on master:


70a30d9 * selftest: sstatetests: skip glibc-initial tests in case of non-glibc distro
6702c7b * selftest/imagefeatures: skip weston test if required features are not present