13803 2020-02-19 20:24:04 +0000 devtool setupClass file copying race 2020-09-30 14:26:57 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified x86 Multiple RESOLVED FIXED backport 3.1 AB-INT Medium+ normal 3.2 M4 1 richard.purdie tim.orling randy.macleod ross.burton tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 86491 0 richard.purdie 2020-02-19 20:24:04 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/704 2020-02-19 20:07:20,543 - oe-selftest - INFO - ====================================================================== 2020-02-19 20:07:20,709 - oe-selftest - INFO - ERROR: setUpModule (devtool) [ (subunit.RemotedTestCase) 2020-02-19 20:07:20,710 - oe-selftest - INFO - ---------------------------------------------------------------------- 2020-02-19 20:07:20,710 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 67, in setUpModule bb.utils.edit_bblayers_conf(bblayers_conf, None, None, bblayers_edit_cb) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1461, in edit_bblayers_conf (updated, newlines) = edit_metadata(newlines, ['BBLAYERS'], handle_bblayers) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1291, in edit_metadata if handle_var_end(): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1216, in handle_var_end (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op, newlines) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1426, in handle_bblayers res = edit_cb(layer, canonicalise_path(layer)) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 59, in bblayers_edit_cb shutil.copytree(pth, destdir) File "/opt/poky/3.0.1/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/shutil.py", line 368, in copytree raise Error(errors) shutil.Error: [('/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.139811433930384', '/tmp/devtoolqa2ey8u4_m/core-copy/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.139811433930384', "[Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.139811433930384'"), ('/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.140620172132720', '/tmp/devtoolqa2ey8u4_m/core-copy/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.140620172132720', "[Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.140620172132720'")] Files are being acessed/created whilst the copy takes place leading to failures. 86526 1 randy.macleod 2020-02-20 16:00:34 +0000 Only seen once so far. 87854 2 tim.orling 2020-07-30 07:28:33 +0000 *** Bug 13421 has been marked as a duplicate of this bug. *** 87872 3 tim.orling 2020-08-03 09:14:18 +0000 We've seen this same pattern in 13803, so more than once. If it is indeed a race condition with .pyc files being used while they are being copied, then the ignore_pattern argument to copytree looks to be the solution [1]: """ Another example that uses the ignore_patterns() helper: from shutil import copytree, ignore_patterns copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*')) This will copy everything except .pyc files and files or directories whose name starts with tmp. """ [1] https://docs.python.org/3/library/shutil.html#copytree-example 87873 4 tim.orling 2020-08-03 09:22:57 +0000 The answers in this SO look promising: https://stackoverflow.com/questions/7842044/filter-directory-when-using-shutil-copytree 88241 5 tim.orling 2020-09-25 10:38:57 +0000 subunit.RemotedTestCase Tells us the offending code (shutil.copytree) only occurs when running oe-selftest in parallel (-j), e.g. oe-selftest -v -j 8 -r devtool Note this requires python3-testtools and python3-subunit installed on the host (need to double check docs are calling that out). 88258 6 tim.orling 2020-09-28 21:42:50 +0000 Yes, it's a one-liner solution, but testing it is non-trivial. oe-selftest assumes you have a large /tmp partition (which I do not), so I had to hack a local branch to use my large build partition. Also, as noted in previous comment, the code in question is only exercised in parallel oe-selftest operation. With HIGHLY instrumented code, I have found no failures of the simple one-liner patch (other than the npm test case which is unrelated). I have also not found a smoking gun that proves I fixed it. But it **should** be the right fix. So, submit and move on to the next bug. 88276 7 tim.orling 2020-09-30 14:26:57 +0000 Merged to master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=31d3042487c7e1d61bcde37801d17e518ca88da0