12084 2017-09-14 05:54:39 +0000 distutils/setuptools recipes may try to fetch code during do_compile 2018-07-31 02:25:27 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 2.4 x86 Multiple RESOLVED FIXED Backport to YP 2.4 and 2.5 Medium+ normal 2.6 M3 1 bluelightning hongxu.jia meta.mr.watcher meta.watcher randy.macleod ross.burton --- --- No (bug/feature does not impact docs) oldest_to_newest 76816 0 bluelightning 2017-09-14 05:54:39 +0000 Recently I have dealt with failures to build two recipes (python-paho-mqtt in meta-python and python-restrictedpython in meta-homeassistant, plus python3 variants of both) where if you did not have pytest-runner installed on your host then it attempted to use pip to download it, and since I happened to be building these on a machine where a proxy was required, this failed as the proxy variables aren't set in the do_compile environment and thus the connection was blocked. If I'd built these on a machine which had direct access to the internet I suspect the issue wouldn't be apparent. Adding an explicit DEPENDS on python(3)-pytest-runner-native fixed the problem: http://cgit.openembedded.org/meta-openembedded/commit?id=2951fb898e7c761e3071521b58c9aa6cc746fba3 Proxy issues aside, we want full control over what gets downloaded during a build, so we definitely don't want to be downloading anything outside of do_fetch. Thus we need to prevent pip from being called here - if a dependency is missing then it should fail immediately. (This was the trigger that exposed bug 11978, although that is a separate issue.) 76839 1 ross.burton 2017-09-14 15:00:46 +0000 Did a few minutes of digging as I was curious: python-paho-mqtt is using setuptools and the setup_requires argument, where setuptools will use pip to fetch the module if it isn't present. Not sure if setuptools has a way to disable this, we may need to patch it into setuptools-native and then pass a special argument to disable fetching. 81212 2 hongxu.jia 2018-07-31 02:25:27 +0000 commit 9d556092fcc6f04c487afd126d52935bac133165 Author: Hongxu Jia <hongxu.jia@windriver.com> Date: Thu Jul 26 22:50:21 2018 +0800 distutils/distutils3: do not try to fetch code during do_configure For distutils3, any setup.py invoking will cause setup_requires argument to trigger a code fetching. Since the following commit applied in oe-core, code fetching occurs during do_confugire before the do_compile. ... b805cef distutils: clean the build tree in do_configure ... Refer what do_compile did, add var-NO_FETCH_BUILD to do_configure. Sync with distutils3, add do_configure to distutils also. commit 57414f4a1c549e6faaa110e6e95ff601b05b7361 Author: Hongxu Jia <hongxu.jia@windriver.com> Date: Tue Jul 17 21:36:35 2018 +0800 distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile If a python recipe is using setuptools and the setup_requires argument, where setuptools will use easy_install to fetch the module if it isn't present. The build failed on a machine where a proxy was required, but succeeded on a machine which had direct access to the internet Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile. Example result: ... ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS. Traceback (most recent call last): File "setup.py", line 56, in <module> ... The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.