| Summary: | Some Python recipes fail when rebuilding sometimes | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Peter Kjellerstedt <peter.kjellerstedt> |
| Component: | devtools / tool chain | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | High | CC: | meta.mr.watcher, meta.watcher, richard.purdie, ross.burton, tim.orling |
| Version: | unspecified | ||
| Target Milestone: | 4.1 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
Got one more: python3-iniparse-native I have actually seen something similar to this, where importlib finds a dependency but doesn't fill in the metadata, so version is None. I only saw it once and didn't preserve a build tree though. If this happens again, can you archive the work directory (including the sysroots) please? I just replicated for python3-setuptools-scm-native. I saved the broken work directory for that recipe, and in a new build tree built the recipe, then compared the trees. They were identical. My hunch is that a path isn't being mangled correctly so something is looking into another recipe's build tree for files, but I may be miles out. I've been digging and the most interesting thing I've found so far is the warnings present in this log which I don't see in my build. Specifically: | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'python_requires' and I do see python_requires being tested in my build. The question is therefore what could be present or broken that could cause such a warning to be emitted... If I comment out the lines:
for ep in metadata.entry_points(group='distutils.setup_keywords'):
vars(self).setdefault(ep.name, None)
in recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py
then I see the warnings as in Peter's log. I also then get an error:
File "/media/build1/poky/build/tmp-test/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/command/build_py.py", line 35, in finalize_options
self.exclude_package_data = self.distribution.exclude_package_data or {}
AttributeError: 'Distribution' object has no attribute 'exclude_package_data'. Did you mean: 'exclude_package'?
which reminds me of a failure seen by Bruce.
There is some horrible intercept/wrapping going on in setuptools of distutils and I think somehow this bug occurs when that wrapping breaks. Still don't know the cause of that though.
To add, the "python_requires" comes from the [distutils.setup_keywords] section of recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools-62.3.1.dist-info/ entry_points.txt. Removing that file causes the code in question to fail to add the argument. The python3-setuptools 59.5.0 to 62.3.1 seems somehow implicated in this since there have been few other changes and docutils has minimal other dependencies. I can confirm that 59.5.0 does have an entry_points file that does mention python_requires. Upgrading my local build does see the old files removed and the new ones moved into place. There are two issues. Python importlib is using the on disk file order to read files and there are empty directories being left behind by previous python modules. If you hit both issues together, your build fails. Patch on the list to tweak importlib. |
When rebuilding after updating some layers, a lot of the python recipes fails with the following error: ERROR: python3-docutils-native-0.18.1-r0 do_compile: 'python3 setup.py bdist_wheel ' execution failed. ERROR: python3-docutils-native-0.18.1-r0 do_compile: ExecutionError('.../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639', 1, None, None) ERROR: Logfile of failure stored in: .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/log.do_compile.20639 Log data follows: | DEBUG: Executing shell function do_compile | + cd .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1 | + do_compile | + setuptools3_do_compile | + cd .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1 | + NO_FETCH_BUILD=1 | + STAGING_INCDIR=.../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/include | + STAGING_LIBDIR=.../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib | + .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/bin/python3-native/python3 setup.py bdist_wheel --verbose --dist-dir .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/dist | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'python_requires' | warnings.warn(msg) | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'include_package_data' | warnings.warn(msg) | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'exclude_package_data' | warnings.warn(msg) | Traceback (most recent call last): | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1/setup.py", line 144, in <module> | do_setup() | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1/setup.py", line 139, in do_setup | dist = setup(**package_data) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup | return distutils.core.setup(**attrs) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 135, in setup | ok = dist.parse_command_line() | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 484, in parse_command_line | args = self._parse_command_opts(parser, args) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py", line 1119, in _parse_command_opts | nargs = _Distribution._parse_command_opts(self, parser, args) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 540, in _parse_command_opts | cmd_class = self.get_command_class(command) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py", line 965, in get_command_class | self._install_dependencies(ep) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py", line 929, in _install_dependencies | for req in nspektr.missing(ep): | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/jaraco/functools.py", line 35, in <lambda> | return lambda *args, **kwargs: f1(f2(*args, **kwargs)) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/contextlib.py", line 79, in inner | return func(*args, **kwds) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/nspektr/__init__.py", line 35, in is_satisfied | return resolve(req) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/nspektr/__init__.py", line 26, in resolve | if not req.specifier.contains(Version(dist.version), prereleases=True): | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/packaging/version.py", line 264, in __init__ | match = self._regex.search(version) | TypeError: expected string or bytes-like object | ERROR: 'python3 setup.py bdist_wheel ' execution failed. | WARNING: .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639:178 exit 1 from 'exit 1' | WARNING: Backtrace (BB generated script): | #1: bbfatal_log, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 178 | #2: setuptools3_do_compile, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 168 | #3: do_compile, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 157 | #4: main, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 183 In this particular build, the following recipes failed: python3-docutils-native python3-mako-native python3-inflection-native python3-setuptools-scm-native python3-packaging-native python3-cython-native python3-pygments-native python3-scons-native python3-pytest-native python3-lark-native python3-sdbus++-native python3-numpy-native python3-six-native