Bug 13803

Summary: devtool setupClass file copying race
Product: [QA/Testing] Functional (self) Testing Reporter: Richard Purdie <richard.purdie>
Component: oe-selftestAssignee: Tim Orling <tim.orling>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: randy.macleod, ross.burton, tim.orling
Version: unspecified   
Target Milestone: 3.2 M4   
Hardware: x86   
OS: Multiple   
Whiteboard: backport 3.1 AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2020-02-19 20:24:04 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/704

2020-02-19 20:07:20,543 - oe-selftest - INFO - ======================================================================
2020-02-19 20:07:20,709 - oe-selftest - INFO - ERROR: setUpModule (devtool) [ (subunit.RemotedTestCase)
2020-02-19 20:07:20,710 - oe-selftest - INFO - ----------------------------------------------------------------------
2020-02-19 20:07:20,710 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 67, in setUpModule
    bb.utils.edit_bblayers_conf(bblayers_conf, None, None, bblayers_edit_cb)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1461, in edit_bblayers_conf
    (updated, newlines) = edit_metadata(newlines, ['BBLAYERS'], handle_bblayers)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1291, in edit_metadata
    if handle_var_end():
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1216, in handle_var_end
    (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op, newlines)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/utils.py", line 1426, in handle_bblayers
    res = edit_cb(layer, canonicalise_path(layer))
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/devtool.py", line 59, in bblayers_edit_cb
    shutil.copytree(pth, destdir)
  File "/opt/poky/3.0.1/sysroots/x86_64-pokysdk-linux/usr/lib/python3.7/shutil.py", line 368, in copytree
    raise Error(errors)
shutil.Error: [('/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.139811433930384', '/tmp/devtoolqa2ey8u4_m/core-copy/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.139811433930384', "[Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.139811433930384'"), ('/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.140620172132720', '/tmp/devtoolqa2ey8u4_m/core-copy/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.140620172132720', "[Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oe/__pycache__/classextend.cpython-37.pyc.140620172132720'")]

Files are being acessed/created whilst the copy takes place leading to failures.
Comment 1 Randy MacLeod 2020-02-20 16:00:34 UTC 
Only seen once so far.
Comment 2 Tim Orling 2020-07-30 07:28:33 UTC 
*** Bug 13421 has been marked as a duplicate of this bug. ***
Comment 3 Tim Orling 2020-08-03 09:14:18 UTC 
We've seen this same pattern in 13803, so more than once.

If it is indeed a race condition with .pyc files being used while they are being copied, then the ignore_pattern argument to copytree looks to be the solution [1]:

"""
Another example that uses the ignore_patterns() helper:

from shutil import copytree, ignore_patterns

copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))
This will copy everything except .pyc files and files or directories whose name starts with tmp.
"""

[1] https://docs.python.org/3/library/shutil.html#copytree-example
Comment 4 Tim Orling 2020-08-03 09:22:57 UTC 
The answers in this SO look promising:

https://stackoverflow.com/questions/7842044/filter-directory-when-using-shutil-copytree
Comment 5 Tim Orling 2020-09-25 10:38:57 UTC 
subunit.RemotedTestCase Tells us the offending code (shutil.copytree) only occurs when running oe-selftest in parallel (-j), e.g.

oe-selftest -v -j 8 -r devtool

Note this requires python3-testtools and python3-subunit installed on the host (need to double check docs are calling that out).
Comment 6 Tim Orling 2020-09-28 21:42:50 UTC 
Yes, it's a one-liner solution, but testing it is non-trivial. oe-selftest assumes you have a large /tmp partition (which I do not), so I had to hack a local branch to use my large build partition. Also, as noted in previous comment, the code in question is only exercised in parallel oe-selftest operation.

With HIGHLY instrumented code, I have found no failures of the simple one-liner patch (other than the npm test case which is unrelated). I have also not found a smoking gun that proves I fixed it. But it **should** be the right fix.

So, submit and move on to the next bug.