15247 2023-10-20 01:54:00 +0000 oe-selftest with --machine=all can fail because BUILDDIR env variable is mismanaged 2023-11-10 14:09:44 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest 5.0 x86 Multiple RESOLVED FIXED Medium+ normal 5.0 M1 1 chris.laplante chris.laplante randy.macleod richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 96750 0 chris.laplante 2023-10-20 01:54:00 +0000 Running oe-selftest with "--machine=all" can fail. Usually for me it fails on the second machine. For simplicity let's just consider the case of running a single test. E.g.: oe-selftest -r devtool.DevtoolAddTests.test_devtool_add_binary --machine all -v setup_builddir overwrites BUILDDIR env variable with the new suffixed build dir, build-st. But it doesn't revert it after the test has been run. So each time it is called (for each machine), it re-suffixes "-st" onto the build dir. You'll see build-st, build-st-st, build-st-st-st, etc. directories get created. The reason for the actual failure is in the code that sets up the meta-selftest layer. The first time around, copytree is performed on: poky/meta-selftest => poky/build-st/meta-selftest And the git repo is initialized. The second time around, BUILDDIR points to 'build-st', so copytree is run as: poky/build-st/meta-selftest => poky/build-st/build-st/meta-selftest The git repo setup is attempted again. The 'git init' succeeds, but the 'git commit' can fail with a '1' return code if no changes are present. Usually, there won't be any changes because we are operating on a git repo that was already setup. But sometimes, new .pyc files can be present which get caught in the 'git add -u'. These will make the 'git commit' succeed. In the run below, I instrumented selftest/context.py to add some additional 'info' messages showing how copytree is called. You'll notice this got through 3 machines before failing. chris@chris-virtual-machine:~/repos/poky$ oe-selftest -r devtool.DevtoolAddTests.test_devtool_add_binary --machine all -v 2023-10-19 21:20:38,702 - oe-selftest - INFO - Custom machine mode enabled. MACHINE set to all 2023-10-19 21:20:38,726 - oe-selftest - INFO - Changing cwd to /home/chris/repos/poky/build 2023-10-19 21:20:38,727 - oe-selftest - INFO - Adding layer libraries: 2023-10-19 21:20:38,727 - oe-selftest - INFO - /home/chris/repos/poky/meta-poky/lib 2023-10-19 21:20:38,727 - oe-selftest - INFO - /home/chris/repos/poky/meta/lib 2023-10-19 21:20:38,727 - oe-selftest - INFO - /home/chris/repos/poky/meta-yocto-bsp/lib 2023-10-19 21:20:38,727 - oe-selftest - INFO - /home/chris/repos/poky/meta-selftest/lib 2023-10-19 21:20:38,727 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2023-10-19 21:20:39,264 - oe-selftest - INFO - Run tests with custom MACHINE set to: qemuppc 2023-10-19 21:20:39,275 - oe-selftest - INFO - copytree: /home/chris/repos/poky/meta-selftest => /home/chris/repos/poky/build-st/meta-selftest hint: Using 'master' as the name for the initial branch. This default branch name hint: is subject to change. To configure the initial branch name to use in all hint: of your new repositories, which will suppress this warning, call: hint: hint: git config --global init.defaultBranch <name> hint: hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and hint: 'development'. The just-created branch can be renamed via this command: hint: hint: git branch -m <name> 2023-10-19 21:20:39,801 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/chris/repos/poky/build-st/conf/local.conf 2023-10-19 21:20:39,801 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-10-19 21:20:41,237 - oe-selftest - INFO - test_devtool_add_binary (devtool.DevtoolAddTests.test_devtool_add_binary) 2023-10-19 21:20:41,240 - oe-selftest - DEBUG - Writing to: /home/chris/repos/poky/build-st/conf/machine.inc MACHINE ??= "qemuppc" 2023-10-19 21:20:41,240 - oe-selftest - DEBUG - Appending to: /home/chris/repos/poky/build-st/conf/selftest.inc SSTATE_DIR = "/home/chris/repos/poky/build-st/sstate_devtool" SSTATE_MIRRORS += "file://.* file:////home/chris/repos/poky/build-st/sstate-cache/PATH" BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" 2023-10-19 21:20:41,241 - oe-selftest - DEBUG - Adding path '/tmp/devtoolqaf9s9rbjc' to be cleaned up when test is over 2023-10-19 21:20:41,249 - oe-selftest - DEBUG - Adding path '/home/chris/repos/poky/build-st/workspace' to be cleaned up when test is over 2023-10-19 21:20:41,249 - oe-selftest - DEBUG - Adding command 'bitbake -c cleansstate tst-bin' to tearDown for this test. 2023-10-19 21:20:41,249 - oe-selftest - DEBUG - Adding command 'bitbake-layers remove-layer */workspace' to tearDown for this test. 2023-10-19 21:36:13,824 - oe-selftest - INFO - ... ok 2023-10-19 21:36:13,824 - oe-selftest - DEBUG - Deleting devtool sstate cache on /home/chris/repos/poky/build-st/sstate_devtool 2023-10-19 21:36:13,956 - oe-selftest - INFO - ---------------------------------------------------------------------- 2023-10-19 21:36:13,956 - oe-selftest - INFO - Ran 1 test in 934.691s 2023-10-19 21:36:13,956 - oe-selftest - INFO - OK 2023-10-19 21:36:17,547 - oe-selftest - INFO - RESULTS: 2023-10-19 21:36:17,547 - oe-selftest - INFO - RESULTS - devtool.DevtoolAddTests.test_devtool_add_binary: PASSED (932.59s) 2023-10-19 21:36:17,550 - oe-selftest - INFO - SUMMARY: 2023-10-19 21:36:17,550 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 934.692s 2023-10-19 21:36:17,550 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) 2023-10-19 21:36:17,555 - oe-selftest - INFO - Changing cwd to /home/chris/repos/poky/build-st 2023-10-19 21:36:17,556 - oe-selftest - INFO - Adding layer libraries: 2023-10-19 21:36:17,556 - oe-selftest - INFO - /home/chris/repos/poky/meta-poky/lib 2023-10-19 21:36:17,556 - oe-selftest - INFO - /home/chris/repos/poky/meta/lib 2023-10-19 21:36:17,556 - oe-selftest - INFO - /home/chris/repos/poky/meta-yocto-bsp/lib 2023-10-19 21:36:17,556 - oe-selftest - INFO - /home/chris/repos/poky/meta-selftest/lib 2023-10-19 21:36:17,557 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2023-10-19 21:36:18,227 - oe-selftest - INFO - Run tests with custom MACHINE set to: qemuarmv5 2023-10-19 21:36:18,244 - oe-selftest - INFO - copytree: /home/chris/repos/poky/build-st/meta-selftest => /home/chris/repos/poky/build-st-st/meta-selftest 2023-10-19 21:36:20,260 - oe-selftest - INFO - test_devtool_add_binary (devtool.DevtoolAddTests.test_devtool_add_binary) 2023-10-19 21:36:20,263 - oe-selftest - DEBUG - Writing to: /home/chris/repos/poky/build-st-st/conf/machine.inc MACHINE ??= "qemuarmv5" 2023-10-19 21:36:20,263 - oe-selftest - DEBUG - Appending to: /home/chris/repos/poky/build-st-st/conf/selftest.inc SSTATE_DIR = "/home/chris/repos/poky/build-st-st/sstate_devtool" SSTATE_MIRRORS += "file://.* file:////home/chris/repos/poky/build-st-st/sstate-cache/PATH" BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" 2023-10-19 21:36:20,264 - oe-selftest - DEBUG - Adding path '/tmp/devtoolqa2jkz7kgw' to be cleaned up when test is over 2023-10-19 21:36:20,274 - oe-selftest - DEBUG - Adding path '/home/chris/repos/poky/build-st-st/workspace' to be cleaned up when test is over 2023-10-19 21:36:20,274 - oe-selftest - DEBUG - Adding command 'bitbake -c cleansstate tst-bin' to tearDown for this test. 2023-10-19 21:36:20,274 - oe-selftest - DEBUG - Adding command 'bitbake-layers remove-layer */workspace' to tearDown for this test. 2023-10-19 21:51:45,995 - oe-selftest - INFO - ... ok 2023-10-19 21:51:45,996 - oe-selftest - DEBUG - Deleting devtool sstate cache on /home/chris/repos/poky/build-st-st/sstate_devtool 2023-10-19 21:51:46,124 - oe-selftest - INFO - ---------------------------------------------------------------------- 2023-10-19 21:51:46,124 - oe-selftest - INFO - Ran 1 test in 927.896s 2023-10-19 21:51:46,124 - oe-selftest - INFO - OK 2023-10-19 21:51:49,572 - oe-selftest - INFO - RESULTS: 2023-10-19 21:51:49,572 - oe-selftest - INFO - RESULTS - devtool.DevtoolAddTests.test_devtool_add_binary: PASSED (925.74s) 2023-10-19 21:51:49,575 - oe-selftest - INFO - SUMMARY: 2023-10-19 21:51:49,575 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 927.897s 2023-10-19 21:51:49,575 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) 2023-10-19 21:51:49,579 - oe-selftest - INFO - Changing cwd to /home/chris/repos/poky/build-st-st 2023-10-19 21:51:49,579 - oe-selftest - INFO - Adding layer libraries: 2023-10-19 21:51:49,579 - oe-selftest - INFO - /home/chris/repos/poky/meta-poky/lib 2023-10-19 21:51:49,579 - oe-selftest - INFO - /home/chris/repos/poky/meta/lib 2023-10-19 21:51:49,579 - oe-selftest - INFO - /home/chris/repos/poky/meta-yocto-bsp/lib 2023-10-19 21:51:49,579 - oe-selftest - INFO - /home/chris/repos/poky/meta-selftest/lib 2023-10-19 21:51:49,580 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2023-10-19 21:51:50,132 - oe-selftest - INFO - Run tests with custom MACHINE set to: qemux86 2023-10-19 21:51:50,149 - oe-selftest - INFO - copytree: /home/chris/repos/poky/build-st-st/meta-selftest => /home/chris/repos/poky/build-st-st-st/meta-selftest Traceback (most recent call last): File "/home/chris/repos/poky/scripts/oe-selftest", line 65, in <module> ret = main() ^^^^^^ File "/home/chris/repos/poky/scripts/oe-selftest", line 49, in main results = args.func(logger, args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 434, in run results.append(self._internal_run(logger, args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 411, in _internal_run rc = self.tc.runTests(**self.tc_kwargs['run']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 177, in runTests return super(OESelftestTestContext, self).runTests(processes, skips) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/core/context.py", line 91, in runTests result = self.runner.run(self.prepareSuite(self.suites, processes)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/.rye/py/cpython@3.12.0/install/lib/python3.12/unittest/runner.py", line 240, in run test(result) File "/home/chris/.rye/py/cpython@3.12.0/install/lib/python3.12/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 47, in run (builddir, newbuilddir) = self.setupfunc("-st", None, self.suite) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 113, in setup_builddir subprocess.check_output("git init && git add * && git commit -a -m 'initial'", cwd=newselftestdir, shell=True) File "/home/chris/.rye/py/cpython@3.12.0/install/lib/python3.12/subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/.rye/py/cpython@3.12.0/install/lib/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'git init && git add * && git commit -a -m 'initial'' returned non-zero exit status 1. 96751 1 chris.laplante 2023-10-20 03:43:21 +0000 Addendum - I was inadvertently running with --keep-temp, which is why it makes it to copytree at all. Here is the new failure with that option turned off. I added instrumentation to removebuilddir to print the directory. chris@chris-virtual-machine:~/repos/poky$ oe-selftest -r devtool.DevtoolAddTests.test_devtool_add_binary --machine all -v 2023-10-19 23:27:01,685 - oe-selftest - INFO - Custom machine mode enabled. MACHINE set to all 2023-10-19 23:27:01,713 - oe-selftest - INFO - Changing cwd to /home/chris/repos/poky/build 2023-10-19 23:27:01,713 - oe-selftest - INFO - Adding layer libraries: 2023-10-19 23:27:01,714 - oe-selftest - INFO - /home/chris/repos/poky/meta-poky/lib 2023-10-19 23:27:01,714 - oe-selftest - INFO - /home/chris/repos/poky/meta/lib 2023-10-19 23:27:01,714 - oe-selftest - INFO - /home/chris/repos/poky/meta-yocto-bsp/lib 2023-10-19 23:27:01,714 - oe-selftest - INFO - /home/chris/repos/poky/meta-selftest/lib 2023-10-19 23:27:01,715 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... (23:27:01.720925) NOTE: Reconnecting to bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... (23:27:01.720925) NOTE: Retrying server connection (#1)... (23:27:01.720925) NOTE: Starting bitbake server... 2023-10-19 23:27:02,266 - oe-selftest - INFO - Run tests with custom MACHINE set to: qemuppc 2023-10-19 23:27:02,281 - oe-selftest - INFO - copytree: /home/chris/repos/poky/meta-selftest => /home/chris/repos/poky/build-st/meta-selftest hint: Using 'master' as the name for the initial branch. This default branch name hint: is subject to change. To configure the initial branch name to use in all hint: of your new repositories, which will suppress this warning, call: hint: hint: git config --global init.defaultBranch <name> hint: hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and hint: 'development'. The just-created branch can be renamed via this command: hint: hint: git branch -m <name> 2023-10-19 23:27:02,779 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/chris/repos/poky/build-st/conf/local.conf 2023-10-19 23:27:02,779 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2023-10-19 23:27:04,053 - oe-selftest - INFO - test_devtool_add_binary (devtool.DevtoolAddTests.test_devtool_add_binary) 2023-10-19 23:27:04,055 - oe-selftest - DEBUG - Writing to: /home/chris/repos/poky/build-st/conf/machine.inc MACHINE ??= "qemuppc" 2023-10-19 23:27:04,056 - oe-selftest - DEBUG - Appending to: /home/chris/repos/poky/build-st/conf/selftest.inc SSTATE_DIR = "/home/chris/repos/poky/build-st/sstate_devtool" SSTATE_MIRRORS += "file://.* file:////home/chris/repos/poky/build-st/sstate-cache/PATH" BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" 2023-10-19 23:27:04,056 - oe-selftest - DEBUG - Adding path '/tmp/devtoolqaq4lyiji7' to be cleaned up when test is over 2023-10-19 23:27:04,064 - oe-selftest - DEBUG - Adding path '/home/chris/repos/poky/build-st/workspace' to be cleaned up when test is over 2023-10-19 23:27:04,064 - oe-selftest - DEBUG - Adding command 'bitbake -c cleansstate tst-bin' to tearDown for this test. 2023-10-19 23:27:04,065 - oe-selftest - DEBUG - Adding command 'bitbake-layers remove-layer */workspace' to tearDown for this test. 2023-10-19 23:41:12,747 - oe-selftest - INFO - ... ok 2023-10-19 23:41:12,747 - oe-selftest - DEBUG - Deleting devtool sstate cache on /home/chris/repos/poky/build-st/sstate_devtool 2023-10-19 23:41:12,877 - oe-selftest - INFO - removing: /home/chris/repos/poky/build-st 2023-10-19 23:41:15,234 - oe-selftest - INFO - ---------------------------------------------------------------------- 2023-10-19 23:41:15,235 - oe-selftest - INFO - Ran 1 test in 852.967s 2023-10-19 23:41:15,235 - oe-selftest - INFO - OK 2023-10-19 23:41:19,277 - oe-selftest - INFO - RESULTS: 2023-10-19 23:41:19,277 - oe-selftest - INFO - RESULTS - devtool.DevtoolAddTests.test_devtool_add_binary: PASSED (848.69s) 2023-10-19 23:41:19,280 - oe-selftest - INFO - SUMMARY: 2023-10-19 23:41:19,280 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 852.968s 2023-10-19 23:41:19,280 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) 2023-10-19 23:41:19,284 - oe-selftest - INFO - Changing cwd to /home/chris/repos/poky/build-st Traceback (most recent call last): File "/home/chris/repos/poky/scripts/oe-selftest", line 65, in <module> ret = main() ^^^^^^ File "/home/chris/repos/poky/scripts/oe-selftest", line 49, in main results = args.func(logger, args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 433, in run results.append(self._internal_run(logger, args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 410, in _internal_run self._pre_run() File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 342, in _pre_run _check_presence_meta_selftest() File "/home/chris/repos/poky/meta/lib/oeqa/selftest/context.py", line 314, in _check_presence_meta_selftest os.chdir(builddir) FileNotFoundError: [Errno 2] No such file or directory: '/home/chris/repos/poky/build-st' 97000 2 randy.macleod 2023-10-26 14:33:28 +0000 Should we remove this machine option entirely? (Richard) YP bug review. 97035 3 richard.purdie 2023-10-29 10:11:46 +0000 Just to be clear, I'm wondering if these machine options for selftest are really needed/useful. If we wanted to do this we'd do it at the autoibuilder level these days. I'm therefore thinking it could be removed. 97244 4 richard.purdie 2023-11-10 14:09:44 +0000 Dropped in https://git.yoctoproject.org/poky/commit/?id=e598223e0b765c0a418c4f036927ef3e567c56ed