Created attachment 4400 [details] log file 2018-12-07 23:50:08,880 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolTests.test_recipetool_load_plugin - Testcase 1640: FAILED (4.00s) https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/58
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/82
*** Bug 13371 has been marked as a duplicate of this bug. ***
*** Bug 13422 has been marked as a duplicate of this bug. ***
After looking at the logs and the codes, I guess this is the similar problem that devtool once encountered. Maybe we should do something similar, copying layers in setUpModule in recipetool.py, just like the following commit does for devtool.py. commit d9a9a2d80e648a9f0556f6e3eb3eea7ee65539ab Author: Paul Eggleton <paul.eggleton@linux.intel.com> Date: Thu Jul 26 16:27:44 2018 +0200 oe-selftest: devtool: avoid parallel races by using temporary copy of core Some of the devtool tests make changes to files under meta/ - legitimately since we want these tests to be working with real recipes and associated files. Unfortunately with the new oe-selftest parallelisation this can break other tests if files go missing at the wrong time (among other scenarios). To avoid this issue, simply take a copy of the core repository and use that for these tests. (We copy the entire repository since changing the path of meta/ influences COREBASE and thus we need to have things like scripts/ alongside as well). (From OE-Core rev: 2457cd57b4195924ef127f497efa2f34f411e660) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hi All, Unfortunately, I could not find the root cause of this problem. It seems that lib/recipetool/bbpath.py is removed from meta-poky layer, causing `recipetool bbdir' not outputing meta-poky path. But checking the codes of selftest cases, I cannot find any one that might remove it. Most `git rm/reset/checkout' commands are done by devtool.py test case, but none of them would remove things from meta-poky. Does anyone have some idea about this failure?
The code in this test adds multiple: lib/recipetool/bbpath.py files in locations from BBPATH and in scripts: poky/meta-poky poky/build poky/meta poky/meta-yocto-bsp poky/meta-selftest poky/scripts I think I'd be tempted to create these files but make them not checked in and not writable by the test user, the run the selftests and see if anything is removing them (or trying to) For this failure to occur, one of the tests must be, we just need to track down which one. Rather than making them non-writable, we could also echo the name of the currently executing test to a file and setup inotify watches on the files, see which test touches them? It would take a while to run but could be done in the background whilst other things are done and come back to it once its done.
still failing: https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/756
I think I finally figured this out. It can be reproduced with a patch along the lines of: diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index 7d8f8950459..06509bc0d3e 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -17,6 +17,7 @@ from oeqa.utils.commands import get_bb_vars, runqemu, get_test_layer oldmetapath = None def setUpModule(): + return import bb.utils global templayerdir @@ -77,7 +78,7 @@ def tearDownModule(): return layerpath bblayers_conf = os.path.join(os.environ['BUILDDIR'], 'conf', 'bblayers.conf') bb.utils.edit_bblayers_conf(bblayers_conf, None, None, bblayers_edit_cb) - shutil.rmtree(templayerdir) + #shutil.rmtree(templayerdir) class DevtoolBase(OESelftestTestCase): diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 6bfe8f177f2..159cfc4ab24 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -74,8 +74,8 @@ class RecipetoolTests(RecipetoolBase): def setUpClass(cls): super(RecipetoolTests, cls).setUpClass() # Ensure we have the right data in shlibs/pkgdata - cls.logger.info('Running bitbake to generate pkgdata') - bitbake('-c packagedata base-files coreutils busybox selftest-recipetool-appendfile') + #cls.logger.info('Running bitbake to generate pkgdata') + #bitbake('-c packagedata base-files coreutils busybox selftest-recipetool-appendfile') bb_vars = get_bb_vars(['COREBASE', 'BBPATH']) cls.corebase = bb_vars['COREBASE'] cls.bbpath = bb_vars['BBPATH'] @@ -554,13 +554,15 @@ class RecipetoolTests(RecipetoolBase): self.assertIn('meta-selftest', srcfile, 'wrong bbpath plugin found') for path in searchpath: self._copy_file_with_cleanup(srcfile, path, 'lib', 'recipetool') + import time + time.sleep(5) result = runCmd("recipetool --quiet count") self.assertEqual(result.output, '1') result = runCmd("recipetool --quiet multiloaded") self.assertEqual(result.output, "no") for path in searchpath: result = runCmd("recipetool --quiet bbdir") - self.assertEqual(result.output, path) + self.assertEqual(result.output, path, msg = "%s != %s (search path is %s)" % (result.output, path, searchpath)) os.unlink(os.path.join(result.output, 'lib', 'recipetool', 'bbpath.py')) finally: with open(srcfile, 'w') as fh: and running: oe-selftest -r recipetool.RecipetoolTests.test_recipetool_load_plugin devtool.DevtoolUpgradeTests.test_devtool_load_plugin -j 2 The patch removes much of the startup overhead of these test classes and causes the plugins to race against each other with one taking a long pause in the middle. The self.track_for_cleanup(dstdir) in _copy_file_with_cleanup() for the "lib" directory is the issue since anything under this directory is also deleted and could belong to the other test. This happens at class teardown time which is consistent with the logs from failing builds.
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e855b384f37e0d781a80fcd65daa23b676e4efac