Bug 12084

Summary: distutils/setuptools recipes may try to fetch code during do_compile
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Paul Eggleton <bluelightning>
Component: devtools / tool chainAssignee: Hongxu Jia <hongxu.jia>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, randy.macleod, ross.burton
Version: 2.4   
Target Milestone: 2.6 M3   
Hardware: x86   
OS: Multiple   
Whiteboard: Backport to YP 2.4 and 2.5
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Paul Eggleton 2017-09-14 05:54:39 UTC 
Recently I have dealt with failures to build two recipes (python-paho-mqtt in meta-python and python-restrictedpython in meta-homeassistant, plus python3 variants of both) where if you did not have pytest-runner installed on your host then it attempted to use pip to download it, and since I happened to be building these on a machine where a proxy was required, this failed as the proxy variables aren't set in the do_compile environment and thus the connection was blocked. If I'd built these on a machine which had direct access to the internet I suspect the issue wouldn't be apparent. Adding an explicit DEPENDS on python(3)-pytest-runner-native fixed the problem:

http://cgit.openembedded.org/meta-openembedded/commit?id=2951fb898e7c761e3071521b58c9aa6cc746fba3

Proxy issues aside, we want full control over what gets downloaded during a build, so we definitely don't want to be downloading anything outside of do_fetch. Thus we need to prevent pip from being called here - if a dependency is missing then it should fail immediately.

(This was the trigger that exposed bug 11978, although that is a separate issue.)
Comment 1 Ross Burton 2017-09-14 15:00:46 UTC 
Did a few minutes of digging as I was curious: python-paho-mqtt is using setuptools and the setup_requires argument, where setuptools will use pip to fetch the module if it isn't present.  Not sure if setuptools has a way to disable this, we may need to patch it into setuptools-native and then pass a special argument to disable fetching.
Comment 2 Hongxu Jia 2018-07-31 02:25:27 UTC 
commit 9d556092fcc6f04c487afd126d52935bac133165
Author: Hongxu Jia <hongxu.jia@windriver.com>
Date:   Thu Jul 26 22:50:21 2018 +0800

    distutils/distutils3: do not try to fetch code during do_configure
    
    For distutils3, any setup.py invoking will cause setup_requires
    argument to trigger a code fetching. Since the following commit
    applied in oe-core, code fetching occurs during do_confugire
    before the do_compile.
    ...
    b805cef distutils: clean the build tree in do_configure
    ...
    
    Refer what do_compile did, add var-NO_FETCH_BUILD to do_configure.
    
    Sync with distutils3, add do_configure to distutils also.
    


commit 57414f4a1c549e6faaa110e6e95ff601b05b7361
Author: Hongxu Jia <hongxu.jia@windriver.com>
Date:   Tue Jul 17 21:36:35 2018 +0800

    distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile
    
    If a python recipe is using setuptools and the setup_requires argument, where
    setuptools will use easy_install to fetch the module if it isn't
    present.
    
    The build failed on a machine where a proxy was required, but succeeded on a
    machine which had direct access to the internet
    
    Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
    allow to fetch code from internet during do_compile.
    
    Example result:
    ...
    ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
    Traceback (most recent call last):
      File "setup.py", line 56, in <module>
    ...
    
    The improvement is flexible for test_requirements argument (used at
    `setup.py test'), where use easy_install also.