When installing packages that use .egg files, such as python3-pip or python3-setuptools, these cannot be imported when opening a python3 interpreter/ pip3 for example is installed using an egg file: # ls /usr/lib/python3.5/site-packages/ | grep pip pip-6.1.1-py3.5.egg But, when trying to import it, the following error is shown: Python 3.5.0 (default, Nov 11 2015, 00:30:30) [GCC 5.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pip Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'pip' We can see that the .egg file isn't part of sys.path: >>> import sys >>> print (sys.path) ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-linux', '/usr/lib/python3.5/lib-dynload', '/usr/lib/python3.5/site-packages', '/usr/lib/python3.5/site-packages/setuptools-18.2-py3.5.egg'] but, if we add it, we're now able to import pip without trouble: >>> sys.path.append('/usr/lib/python3.5/site-packages/pip-6.1.1-py3.5.egg') >>> import pip >>> We need to add a .pth file to /usr/lib/python3.5/site-packages/ containing the path to the installed egg file, like python3-setuptools package does. I suggest we should add this functionality to the distutils3.bbclass, so this doesn't have to be done by every recipe, its likely this affects distutils on python2 as well, we should check that too.
Should we ship packages that contain eggs? The python 2 setuptools code disables eggs and packages get installed with the egg metadata but not inside an actual egg.
(In reply to comment #1) > Should we ship packages that contain eggs? The python 2 setuptools code > disables eggs and packages get installed with the egg metadata but not > inside an actual egg. I would agree, I don't see any reason to install them as eggs, but the code to disable this behavior is there on the setuptools3.bbclass too, perhaps when setuptools was merged with distribute upstream this option was deprecated, there might be another way of doing it, it's definitely worth checking
(In reply to comment #2) > (In reply to comment #1) > > Should we ship packages that contain eggs? The python 2 setuptools code > > disables eggs and packages get installed with the egg metadata but not > > inside an actual egg. > > I would agree, I don't see any reason to install them as eggs, but the code > to disable this behavior is there on the setuptools3.bbclass too, perhaps > when setuptools was merged with distribute upstream this option was > deprecated, there might be another way of doing it, it's definitely worth > checking Actually the code its working just fine, it is extracting the metadata correctly, as we can see here, the pip-6.1.1-py3.5.egg is a directory, and it contains the metadata. root@qemux86:/usr/lib/python3.5/site-packages# ls -l total 360 -rw-r--r-- 1 root root 119 Nov 11 21:07 README drwxr-xr-x 2 root root 1024 Nov 12 18:19 __pycache__ drwxr-xr-x 4 root root 1024 Nov 12 18:20 pip-6.1.1-py3.5.egg -rw-r--r-- 1 root root 361195 Nov 12 18:19 setuptools-18.2-py3.5.egg -rw-r--r-- 1 root root 28 Nov 12 18:19 setuptools.pth -rw-r--r-- 1 root root 2389 Nov 12 18:19 site.py So, apparently the problem is that when checking sys.path it doesn't work recursively, hence why pip in this case, cannot be imported.
Tried to reproduce this on other python modules from meta, but the only two recipes that presented this issue were the ones mentioned here (pip & setuptools). These two cases were already fixed in the specific recipes for pip & setuptools. Thought about moving this code to distutils, but I believe that it is better as it is right now since that alternative only puts unneeded files for modules that do not show this problem. Made a wider search with a great number of python modules from all kind of places and only saw this problem in 2 more modules, but those are out of this scope since they belong to the meta-openembedded layer. These two modules were python-pyparsing and python-simplejson. I'm closing this for now since there is no work to do in the meta/ folder. Will keep an eye open for this happening in some other recipe.
I can confirm I see this same issue on python-numpy: root@qemux86:~# python Python 2.7.11 (default, Jun 6 2016, 18:29:07) [GCC 6.1.1 20160511] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named numpy and again, doing: root@qemux86:~# echo /usr/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg > /usr/lib/python2.7/site-packages/numpy.pth Fixes the problem root@qemux86:~# python Python 2.7.11 (default, Jun 6 2016, 18:29:07) [GCC 6.1.1 20160511] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> This should be done automatically from the distutils class.
Was this fixed by dd756bb8fdff676dba312e38ebc90c4c89dfe855 or was that just a workaround?
It was a workaround only for python-numpy,it could still happen to any package using distutils, I changed most of them to setuptools for now though.
My fault, moving it to M1 again. CCB should agree on any milestone movement.
This is not affecting master branch. Moving it to M2.
This is an enhancement and due to time constrains, it won't be ready for M3. Moving it to first milestone for next release (2.3).
I've pushed a patch to poky-contrib at: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/bug8673&id=80014962b523005717d1a562cca77849fa4f551b please provide feedback.
(In reply to comment #11) > I've pushed a patch to poky-contrib at: > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/ > bug8673&id=80014962b523005717d1a562cca77849fa4f551b > > please provide feedback. While I believe the naming of the egg file will always comply with what you wrote on your patch, I still think a better way would be to find the name of the creted egg file instead of checking for the name we assume it'll have, and then add that path to the pth file, also, it should be worth checking if this happens on python2 as well (distutils class), I see that python-numpy (py2) had issues but I can't recall exactly
Alex, I've pushed a revised version to contrib. please provide feedback: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/bug8673 Also, I did test python-setuptools and python-numpy but this behavior was not detected.
(In reply to comment #13) > Alex, I've pushed a revised version to contrib. please provide feedback: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lamego/ > bug8673 > > Also, I did test python-setuptools and python-numpy but this behavior was > not detected. Looks good in my opinion, the only thing I would do is separate into 3 or at least 2 patches, one with the change to the distutils class, and the other(s) fixing the recipes after the class has that functionality
Patches sent to the mailing list can be seen at: https://patchwork.openembedded.org/series/4820/
I believe the patches have not landed to master but will be soon at M3.
A reminder was sent to the mailing list. Waiting for comments/merging.
A reminder was sent to the ML. Moving to 2.4, waiting for merging.
The created .egg files/directories are now being appropriately added to PATH, allowing both the python3-pip and python3-setuptools modules to be imported at the python3 interpreter. Reported behavior cannot be reproduced, so I'm closing this as not a bug.