8673 2015-11-11 06:47:48 +0000 distutils3 packages installed via an egg file cannot be imported by the python3 interpreter 2017-05-29 17:04:54 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 2.0 x86 Multiple RESOLVED NOTABUG Medium+ enhancement 2.4 M1 1 alejandro.hernandez jose.a.lamego leonardo.sandoval.gonzalez meta.mr.watcher meta.watcher ross.burton --- --- No (bug/feature does not impact docs) oldest_to_newest 56373 0 alejandro.hernandez 2015-11-11 06:47:48 +0000 When installing packages that use .egg files, such as python3-pip or python3-setuptools, these cannot be imported when opening a python3 interpreter/ pip3 for example is installed using an egg file: # ls /usr/lib/python3.5/site-packages/ | grep pip pip-6.1.1-py3.5.egg But, when trying to import it, the following error is shown: Python 3.5.0 (default, Nov 11 2015, 00:30:30) [GCC 5.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pip Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'pip' We can see that the .egg file isn't part of sys.path: >>> import sys >>> print (sys.path) ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-linux', '/usr/lib/python3.5/lib-dynload', '/usr/lib/python3.5/site-packages', '/usr/lib/python3.5/site-packages/setuptools-18.2-py3.5.egg'] but, if we add it, we're now able to import pip without trouble: >>> sys.path.append('/usr/lib/python3.5/site-packages/pip-6.1.1-py3.5.egg') >>> import pip >>> We need to add a .pth file to /usr/lib/python3.5/site-packages/ containing the path to the installed egg file, like python3-setuptools package does. I suggest we should add this functionality to the distutils3.bbclass, so this doesn't have to be done by every recipe, its likely this affects distutils on python2 as well, we should check that too. 56433 1 ross.burton 2015-11-12 16:00:42 +0000 Should we ship packages that contain eggs? The python 2 setuptools code disables eggs and packages get installed with the egg metadata but not inside an actual egg. 56446 2 alejandro.hernandez 2015-11-12 17:47:07 +0000 (In reply to comment #1) > Should we ship packages that contain eggs? The python 2 setuptools code > disables eggs and packages get installed with the egg metadata but not > inside an actual egg. I would agree, I don't see any reason to install them as eggs, but the code to disable this behavior is there on the setuptools3.bbclass too, perhaps when setuptools was merged with distribute upstream this option was deprecated, there might be another way of doing it, it's definitely worth checking 56451 3 alejandro.hernandez 2015-11-12 21:53:13 +0000 (In reply to comment #2) > (In reply to comment #1) > > Should we ship packages that contain eggs? The python 2 setuptools code > > disables eggs and packages get installed with the egg metadata but not > > inside an actual egg. > > I would agree, I don't see any reason to install them as eggs, but the code > to disable this behavior is there on the setuptools3.bbclass too, perhaps > when setuptools was merged with distribute upstream this option was > deprecated, there might be another way of doing it, it's definitely worth > checking Actually the code its working just fine, it is extracting the metadata correctly, as we can see here, the pip-6.1.1-py3.5.egg is a directory, and it contains the metadata. root@qemux86:/usr/lib/python3.5/site-packages# ls -l total 360 -rw-r--r-- 1 root root 119 Nov 11 21:07 README drwxr-xr-x 2 root root 1024 Nov 12 18:19 __pycache__ drwxr-xr-x 4 root root 1024 Nov 12 18:20 pip-6.1.1-py3.5.egg -rw-r--r-- 1 root root 361195 Nov 12 18:19 setuptools-18.2-py3.5.egg -rw-r--r-- 1 root root 28 Nov 12 18:19 setuptools.pth -rw-r--r-- 1 root root 2389 Nov 12 18:19 site.py So, apparently the problem is that when checking sys.path it doesn't work recursively, hence why pip in this case, cannot be imported. 61662 4 humberto.ibarra.lopez 2016-04-25 20:57:02 +0000 Tried to reproduce this on other python modules from meta, but the only two recipes that presented this issue were the ones mentioned here (pip & setuptools). These two cases were already fixed in the specific recipes for pip & setuptools. Thought about moving this code to distutils, but I believe that it is better as it is right now since that alternative only puts unneeded files for modules that do not show this problem. Made a wider search with a great number of python modules from all kind of places and only saw this problem in 2 more modules, but those are out of this scope since they belong to the meta-openembedded layer. These two modules were python-pyparsing and python-simplejson. I'm closing this for now since there is no work to do in the meta/ folder. Will keep an eye open for this happening in some other recipe. 62739 5 alejandro.hernandez 2016-06-07 23:11:37 +0000 I can confirm I see this same issue on python-numpy: root@qemux86:~# python Python 2.7.11 (default, Jun 6 2016, 18:29:07) [GCC 6.1.1 20160511] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named numpy and again, doing: root@qemux86:~# echo /usr/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg > /usr/lib/python2.7/site-packages/numpy.pth Fixes the problem root@qemux86:~# python Python 2.7.11 (default, Jun 6 2016, 18:29:07) [GCC 6.1.1 20160511] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> This should be done automatically from the distutils class. 63179 6 ross.burton 2016-06-21 11:09:42 +0000 Was this fixed by dd756bb8fdff676dba312e38ebc90c4c89dfe855 or was that just a workaround? 63222 7 alejandro.hernandez 2016-06-21 20:02:38 +0000 It was a workaround only for python-numpy,it could still happen to any package using distutils, I changed most of them to setuptools for now though. 63305 8 leonardo.sandoval.gonzalez 2016-06-23 19:48:36 +0000 My fault, moving it to M1 again. CCB should agree on any milestone movement. 63547 9 leonardo.sandoval.gonzalez 2016-06-30 20:19:44 +0000 This is not affecting master branch. Moving it to M2. 65515 10 leonardo.sandoval.gonzalez 2016-08-25 17:13:54 +0000 This is an enhancement and due to time constrains, it won't be ready for M3. Moving it to first milestone for next release (2.3). 69805 11 jose.a.lamego 2017-01-13 17:28:05 +0000 I've pushed a patch to poky-contrib at: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/bug8673&id=80014962b523005717d1a562cca77849fa4f551b please provide feedback. 69806 12 alejandro.hernandez 2017-01-13 18:07:33 +0000 (In reply to comment #11) > I've pushed a patch to poky-contrib at: > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/ > bug8673&id=80014962b523005717d1a562cca77849fa4f551b > > please provide feedback. While I believe the naming of the egg file will always comply with what you wrote on your patch, I still think a better way would be to find the name of the creted egg file instead of checking for the name we assume it'll have, and then add that path to the pth file, also, it should be worth checking if this happens on python2 as well (distutils class), I see that python-numpy (py2) had issues but I can't recall exactly 69840 13 jose.a.lamego 2017-01-16 21:08:16 +0000 Alex, I've pushed a revised version to contrib. please provide feedback: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/bug8673 Also, I did test python-setuptools and python-numpy but this behavior was not detected. 69844 14 alejandro.hernandez 2017-01-16 22:56:24 +0000 (In reply to comment #13) > Alex, I've pushed a revised version to contrib. please provide feedback: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/ > bug8673 > > Also, I did test python-setuptools and python-numpy but this behavior was > not detected. Looks good in my opinion, the only thing I would do is separate into 3 or at least 2 patches, one with the change to the distutils class, and the other(s) fixing the recipes after the class has that functionality 69858 15 jose.a.lamego 2017-01-17 15:47:04 +0000 Patches sent to the mailing list can be seen at: https://patchwork.openembedded.org/series/4820/ 70133 16 leonardo.sandoval.gonzalez 2017-01-25 17:14:52 +0000 I believe the patches have not landed to master but will be soon at M3. 70570 17 jose.a.lamego 2017-02-08 22:45:43 +0000 A reminder was sent to the mailing list. Waiting for comments/merging. 71221 18 jose.a.lamego 2017-03-13 20:32:58 +0000 A reminder was sent to the ML. Moving to 2.4, waiting for merging. 73547 19 jose.a.lamego 2017-05-29 17:04:54 +0000 The created .egg files/directories are now being appropriately added to PATH, allowing both the python3-pip and python3-setuptools modules to be imported at the python3 interpreter. Reported behavior cannot be reproduced, so I'm closing this as not a bug.