12076 2017-09-13 10:25:22 +0000 PYTHONPATH pointing to python2.7 site-packages causes bmaptool to crash 2017-09-15 11:06:56 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple RESOLVED FIXED High major 2.4 M4 1 eduard.bartosh joshuagloe infras.ab.watcher Infras.watcher pidge --- --- No (bug/feature does not impact docs) oldest_to_newest 76779 0 eduard.bartosh 2017-09-13 10:25:22 +0000 PYTHONPATH environment variable contains path to python 2.7 site-packages. This makes bmaptool (and probably any other python 3 program) to fail when importing pkg_resources module from python 2.7 site-packages directory: File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/core2-64-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/usr/bin/bmaptool", line 6, in <module> from pkg_resources import load_entry_point File "<frozen importlib._bootstrap>", line 969, in _find_and_load File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 896, in _find_spec File "<frozen importlib._bootstrap_external>", line 1147, in find_spec File "<frozen importlib._bootstrap_external>", line 1123, in _get_spec File "<frozen importlib._bootstrap_external>", line 1104, in _legacy_get_spec File "<frozen importlib._bootstrap>", line 444, in spec_from_loader File "<frozen importlib._bootstrap_external>", line 541, in spec_from_file_location File "/home/pokybuild/yocto-autobuilder/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 29 def _bypass_ensure_directory(name, mode=0777): ^ SyntaxError: invalid token That makes wic test cases using bmaptool to fail on AB: https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/512/steps/Running%20oe-selftest/logs/stdio 76786 1 eduard.bartosh 2017-09-13 14:15:53 +0000 Reproduced on centos 7: PYTHONPATH=/usr/lib/python2.7/site-packages/ python3.6 -c 'from pkg_resources import load_entry_point' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 78 def _bypass_ensure_directory(name, 0777): ^ SyntaxError: invalid token The reason is that octal literals have different prefix in Python 3. Anyway, it's not a good idea to even try to import modules from python 2 in Python 3 script. 76788 2 joshuagloe 2017-09-13 14:27:50 +0000 Early in the implementation of the yocto-autobuilder it was decided that rather than installing the software & its dependencies it would be run directly from a clone of the repository. An environment setup script is used to set variables, including PYTHONPATH, required to support this workflow. The problem therefore is how we support the existing "run from a clone" workflow without polluting the environment with a PYTHONPATH that adds Python 2.7 modules. 76789 3 eduard.bartosh 2017-09-13 14:57:11 +0000 if we run python 3 and use python 3 libraries in PYTHONPATH we still can run from a clone. 76790 4 joshuagloe 2017-09-13 14:59:20 +0000 (In reply to comment #3) > if we run python 3 and use python 3 libraries in PYTHONPATH we still can run > from a clone. That's certainly true, but we can't port yocto-autobuilder to Python 3 in time to stop this failure breaking M4 builds. 76824 5 joshuagloe 2017-09-14 11:18:06 +0000 1) add a .pth file to a location in site.getusersitepackages() i.e. ~/.local/lib/python2.7/site-packages that lists the modules we carry in our repository 2) Update each executable (i.e. bin/twistd) to add entries to sys.path 3) link the autobuilders lib/python2.7/site-packages contents into ~/.local/lib/python2.7/site-packages I'm leaning towards 1 as that's a little easier to track and undo. Any of these options are antithetical to the decision to not have the AB install and affect the host system, but that's not a decision I care to enforce moving forwards. 76829 6 joshuagloe 2017-09-14 12:46:26 +0000 Implementing solution which adds an autobuilder.pth to the site.getusersitepackages() 76831 7 joshuagloe 2017-09-14 13:51:42 +0000 Implemented and in testing on the old cluster. http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder/commit/?id=7c77c01d4818f0cdb9a325a865379bb673b26bcc plus a couple of follow-ups: http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder/commit/?id=4f0334e33e16bb8e407eb2da2e2988caa7d8b1f7 http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder/commit/?id=21f14553d10b60ed7fd926f7fc3f374c32e78e95 76885 8 joshuagloe 2017-09-15 11:06:56 +0000 Changes now deployed to both clusters.