14816 2022-05-25 23:00:28 +0000 Some Python recipes fail when rebuilding sometimes 2022-05-31 12:52:26 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain unspecified x86 Multiple RESOLVED FIXED High normal 4.1 M1 1 peter.kjellerstedt richard.purdie meta.mr.watcher meta.watcher richard.purdie ross.burton tim.orling --- --- Don't know oldest_to_newest 93318 0 peter.kjellerstedt 2022-05-25 23:00:28 +0000 When rebuilding after updating some layers, a lot of the python recipes fails with the following error: ERROR: python3-docutils-native-0.18.1-r0 do_compile: 'python3 setup.py bdist_wheel ' execution failed. ERROR: python3-docutils-native-0.18.1-r0 do_compile: ExecutionError('.../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639', 1, None, None) ERROR: Logfile of failure stored in: .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/log.do_compile.20639 Log data follows: | DEBUG: Executing shell function do_compile | + cd .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1 | + do_compile | + setuptools3_do_compile | + cd .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1 | + NO_FETCH_BUILD=1 | + STAGING_INCDIR=.../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/include | + STAGING_LIBDIR=.../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib | + .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/bin/python3-native/python3 setup.py bdist_wheel --verbose --dist-dir .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/dist | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'python_requires' | warnings.warn(msg) | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'include_package_data' | warnings.warn(msg) | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'exclude_package_data' | warnings.warn(msg) | Traceback (most recent call last): | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1/setup.py", line 144, in <module> | do_setup() | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/docutils-0.18.1/setup.py", line 139, in do_setup | dist = setup(**package_data) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup | return distutils.core.setup(**attrs) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 135, in setup | ok = dist.parse_command_line() | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 484, in parse_command_line | args = self._parse_command_opts(parser, args) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py", line 1119, in _parse_command_opts | nargs = _Distribution._parse_command_opts(self, parser, args) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 540, in _parse_command_opts | cmd_class = self.get_command_class(command) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py", line 965, in get_command_class | self._install_dependencies(ep) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py", line 929, in _install_dependencies | for req in nspektr.missing(ep): | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/jaraco/functools.py", line 35, in <lambda> | return lambda *args, **kwargs: f1(f2(*args, **kwargs)) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/contextlib.py", line 79, in inner | return func(*args, **kwds) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/nspektr/__init__.py", line 35, in is_satisfied | return resolve(req) | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/nspektr/__init__.py", line 26, in resolve | if not req.specifier.contains(Version(dist.version), prereleases=True): | File ".../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_vendor/packaging/version.py", line 264, in __init__ | match = self._regex.search(version) | TypeError: expected string or bytes-like object | ERROR: 'python3 setup.py bdist_wheel ' execution failed. | WARNING: .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639:178 exit 1 from 'exit 1' | WARNING: Backtrace (BB generated script): | #1: bbfatal_log, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 178 | #2: setuptools3_do_compile, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 168 | #3: do_compile, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 157 | #4: main, .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/temp/run.do_compile.20639, line 183 In this particular build, the following recipes failed: python3-docutils-native python3-mako-native python3-inflection-native python3-setuptools-scm-native python3-packaging-native python3-cython-native python3-pygments-native python3-scons-native python3-pytest-native python3-lark-native python3-sdbus++-native python3-numpy-native python3-six-native 93319 1 peter.kjellerstedt 2022-05-26 00:41:49 +0000 Got one more: python3-iniparse-native 93320 2 ross.burton 2022-05-26 09:03:41 +0000 I have actually seen something similar to this, where importlib finds a dependency but doesn't fill in the metadata, so version is None. I only saw it once and didn't preserve a build tree though. If this happens again, can you archive the work directory (including the sysroots) please? 93333 3 ross.burton 2022-05-26 17:14:00 +0000 I just replicated for python3-setuptools-scm-native. I saved the broken work directory for that recipe, and in a new build tree built the recipe, then compared the trees. They were identical. My hunch is that a path isn't being mangled correctly so something is looking into another recipe's build tree for files, but I may be miles out. 93340 4 richard.purdie 2022-05-27 12:36:24 +0000 I've been digging and the most interesting thing I've found so far is the warnings present in this log which I don't see in my build. Specifically: | .../tmp/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'python_requires' and I do see python_requires being tested in my build. The question is therefore what could be present or broken that could cause such a warning to be emitted... 93342 5 richard.purdie 2022-05-27 13:46:23 +0000 If I comment out the lines: for ep in metadata.entry_points(group='distutils.setup_keywords'): vars(self).setdefault(ep.name, None) in recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/dist.py then I see the warnings as in Peter's log. I also then get an error: File "/media/build1/poky/build/tmp-test/work/x86_64-linux/python3-docutils-native/0.18.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools/command/build_py.py", line 35, in finalize_options self.exclude_package_data = self.distribution.exclude_package_data or {} AttributeError: 'Distribution' object has no attribute 'exclude_package_data'. Did you mean: 'exclude_package'? which reminds me of a failure seen by Bruce. There is some horrible intercept/wrapping going on in setuptools of distutils and I think somehow this bug occurs when that wrapping breaks. Still don't know the cause of that though. 93343 6 richard.purdie 2022-05-27 14:10:46 +0000 To add, the "python_requires" comes from the [distutils.setup_keywords] section of recipe-sysroot-native/usr/lib/python3.10/site-packages/setuptools-62.3.1.dist-info/ entry_points.txt. Removing that file causes the code in question to fail to add the argument. 93344 7 richard.purdie 2022-05-27 14:15:07 +0000 The python3-setuptools 59.5.0 to 62.3.1 seems somehow implicated in this since there have been few other changes and docutils has minimal other dependencies. I can confirm that 59.5.0 does have an entry_points file that does mention python_requires. Upgrading my local build does see the old files removed and the new ones moved into place. 93345 8 richard.purdie 2022-05-27 16:13:33 +0000 There are two issues. Python importlib is using the on disk file order to read files and there are empty directories being left behind by previous python modules. If you hit both issues together, your build fails. Patch on the list to tweak importlib. 93357 9 richard.purdie 2022-05-31 12:52:26 +0000 https://git.yoctoproject.org/poky/commit/?id=012f2db86b96b35eaaefe7f9d33dd820e3a520ec