Currently python3 is provided by python3-core, which does not include all of python3 (in my case it was missing the operator module). As discussed on IRC we should create a meta-package which depends on all packages required to have a proper python3 installation. I'm unsure which packages should be included in this meta package, I'm guessing much of python3-modules and python3-misc should be included. I'm also guessing we want to do the same for python2, however I have not investigated that further.
meta-FOO is usually for layer so change title to reflect intended implementation.
Moving to 2.4 due to workload.
By using the CORE_IMAGE_EXTRA_INSTALL variable to customize the python3 packages installed in the final image, it is possible to set the available python3 options. For example, by including CORE_IMAGE_EXTRA_INSTALL = " python3" in local.conf, most of usual python3 modules are available (including the mentioned "operator"). This makes unnecessary to create a new python3 packagegroup. Having said this, I've created and submitted the requested packagegroup to the mailing list, where I expect to gather enough feedback before dismissing or accepting the change: http://lists.openembedded.org/pipermail/openembedded-core/2017-August/140830.html
I have a package which has a dependency on python3 that does not work with a subset of it. I do not necessarily know which or even if any of the packages I include have a transitive dependency on python3, it might even be so that I add such a package long after building the image. My understanding of CORE_IMAGE_EXTRA_INSTALL is that that still doesn't provide a python3 package, but rather just give me the ability to install additional packages that otherwise would not be. So if my understanding is correct then CORE_IMAGE_EXTRA_INSTALL does not solve my problem. To me it even seems like CORE_IMAGE_EXTRA_INSTALL would only work if I know during building all transitive dependencies that will ever be installed on the system, which I cannot. Moreover, as a package developer I expect python3 to be python3, and not a subset of it, that is it violates the principle of least surprise. Thus, with reservations on me being wrong about what CORE_IMAGE_EXTRA_INSTALL actually does, I think having a python3 packagegroup that correspond to the python3 standard library is the solution (if not complementary to CORE_IMAGE_EXTRA_INSTALL = " python3").
I am not expert in this area but here are yy-two-cents: oe-core provides just the basic recipes/packages and perhaps the python3 packages (provided by oe-core) are too granular for your needs. In case you need to group a core set of packages, I believe the best place to in your layer and not in oe-core. One other thing: What it would be nice is to have packagesgroups (in oe-core) by areas (networking, OS, threading, etc.), so it would be easier to pick a group rather than cherry picking each module.
> perhaps the python3 packages (provided by oe-core) are too granular for your needs Well, the problem is not so much granularity as it being incorrect. It makes a lot of sense that python3-core only contains a subset of python-3, especially given the name. However currently the package 'python3' (a virtual package provided by python3-core) is in fact not python-3, which is why I opened this bug.
(In reply to comment #6) > > perhaps the python3 packages (provided by oe-core) are too granular for your needs > > Well, the problem is not so much granularity as it being incorrect. It makes > a lot of sense that python3-core only contains a subset of python-3, > especially given the name. However currently the package 'python3' (a > virtual package provided by python3-core) is in fact not python-3, which is > why I opened this bug. I beieve you are missing the point here, this is python that will run on embedded systems, this is not a complete Linux distribuition, we definitely do not want to install the whole python package (2 or 3) on an embedded system, at least not usually since we want to save up space, thats the reason why OUR python2 and/or python3 packages provide only the bare minimum to be able to run it, so it is expected (and thats the way it should be, for the reeasons provided above) that it wont work with all applications developed in python. Usually the developer should figure out what dependencies it needs in specific, and install them all one by one, to have the bare minimum (again) for a certain application to run, hence why we need the granularity provided by the python manifest on OE-Core. Although, if one is lazy enough or dont want to figure out the applications python dependencies, installing python-modules would solve the problem mentioned above (thats the reason that package exists). Simply skimming through the python3 recipe you can figure out the way this works and what to do: - python3-modules RDEPENDS on python3-misc, and obviously on python3-core - python3-modules RDEPENDS on all python3 packages created by us on the manifest - python3-misc contains everything that is not contained on any other package. So: IMAGE_INSTALL_append = " python3-modules" would install python3-core (bare minimum) + python3-modules (all out python packages) + python3-misc (everything not contained in the other packages). As a summary: - python3 only installs python3-core and that is the correct behavior (for us, this is not a complete Linux distribuition), since that is what we want in most cases for embedded systems/applications. - Every developer should figure out that applications dependencies and install them one by one. - If that cant be done, just install python3-modules and that would install python3's upstream package / the complete package, but this is not recommended. - No meta-package or packagroup is needed since that would solve the "problem".
> I beieve you are missing the point here I think I understand your point, however I don't think I have been able to convey my point, let me try. > this is python that will run on embedded systems, this is not a complete > Linux distribuition This can quickly lead to a semantic discussion about what an embedded system is and I would rather avoid it, instead I will tell you my point of view. We are using yocto in what we consider an embedded system, we depend on python3 and the extra requirements that entails are not problematic for us. > Usually the developer should figure out what dependencies it needs in > specific, and install them all one by one, to have the bare minimum (again) > for a certain application to run, hence why we need the granularity provided > by the python manifest on OE-Core. Yes, this is very sensible and gives users the ability to have a really slim system, which is awesome. > IMAGE_INSTALL_append = " python3-modules" While this would work it would also introduce a lot of dependencies that we in fact know we don't need, making our system larger than needed. > Every developer should figure out that applications dependencies and install > them one by one. That would be really nice, but sadly we prioritize other things than a super slim final system (perhaps this makes yocto unsuitable for us, I don't really know, I would like to think it is suitable). Basically, we develop against upstream python3 and the python3 standard library, thus we expected the python3 package to work for us, but it did not, as the python3 recipe is not in fact python3. This also quickly becomes very tricky with third party libraries that also assume a full python3 environment. In fact I don't even know how I would do that. pip can tell me what libraries I need, but not which part of python I need. > If that cant be done, just install python3-modules and that would install > python3's upstream package / the complete package, but this is not > recommended This would work, for sure, but my understanding is that python3-modules is a strict superset of python3. > python3 only installs python3-core and that is the correct behavior (for us, > this is not a complete Linux distribuition), since that is what we want in > most cases for embedded systems/applications. I'm taking this last as its the core of my argument. python3-core is not python3 and that is confusing and have caused me quite the headache. I think that everything that's part of python3 is already in oe-core, its just not bundled as python3 currently. Lets take the patch sent by Jose Lamego[1] and call it python3-full for simplicity. I suggest one of the two solutions: 1. Accept python3-full and let it provide python3 2. Accept python3-full and let python3-core still provide python3. I would argue solution 1. is the more correct one, however it would increase the size of artifacts for users who depend on python3 when they in fact should depend on python3-core, which I think is part of your argument. Solution 2. would still solve the problem. It would have been very clear to me and users what I need to get python3 and its entire library. Thus, if I understand your point correctly you are against solution 1. which makes sense, however I don't see any reason for then not going with solution 2. [1]: http://lists.openembedded.org/pipermail/openembedded-core/2017-August/140830.html
(In reply to comment #8) > > I beieve you are missing the point here > > I think I understand your point, however I don't think I have been able to > convey my point, let me try. > > > this is python that will run on embedded systems, this is not a complete > > Linux distribuition > > This can quickly lead to a semantic discussion about what an embedded system > is and I would rather avoid it, instead I will tell you my point of view. We > are using yocto in what we consider an embedded system, we depend on python3 > and the extra requirements that entails are not problematic for us. I completely understand your point, and I know that this is not problematic for YOUR specific case, but it is problematic most cases. > > > Usually the developer should figure out what dependencies it needs in > > specific, and install them all one by one, to have the bare minimum (again) > > for a certain application to run, hence why we need the granularity provided > > by the python manifest on OE-Core. > > Yes, this is very sensible and gives users the ability to have a really slim > system, which is awesome. > > > IMAGE_INSTALL_append = " python3-modules" > > While this would work it would also introduce a lot of dependencies that we > in > fact know we don't need, making our system larger than needed. This would do exactly what you are stating you want, the full python3 package, and literally you just mentioned that extra requirements are nor problematic for you. > > > Every developer should figure out that applications dependencies and install > > them one by one. > > That would be really nice, but sadly we prioritize other things than a super > slim final system (perhaps this makes yocto unsuitable for us, I don't really > know, I would like to think it is suitable). > > Basically, we develop against upstream python3 and the python3 standard > library, thus we expected the python3 package to work for us, but it did not, > as the python3 recipe is not in fact python3. Again, python3-modules is what you want. > > This also quickly becomes very tricky with third party libraries that also > assume a full python3 environment. In fact I don't even know how I would do > that. pip can tell me what libraries I need, but not which part of python I > need. > > > If that cant be done, just install python3-modules and that would install > > python3's upstream package / the complete package, but this is not > > recommended > > This would work, for sure, but my understanding is that python3-modules is a > strict superset of python3. python3-modules is not a superset of python3, like I explained on my comment above, installing python3-modules would end up creating an image with what you want, which is python3 from upstream , nothing extra, nothing less. > > > python3 only installs python3-core and that is the correct behavior (for us, > > this is not a complete Linux distribuition), since that is what we want in > > most cases for embedded systems/applications. > > I'm taking this last as its the core of my argument. python3-core is not > python3 and that is confusing and have caused me quite the headache. Sorry but python3 IS python3-core for OUR CASE, not saying that it is for any other distro, maybe for you it seems unreasonable, but this is an open source project and what is reasonable for most people is what should be installed, for most people using Yocto python3-core IS what should be installed when requesting python3. > > I think that everything that's part of python3 is already in oe-core, its > just > not bundled as python3 currently. Lets take the patch sent by Jose Lamego[1] > and call it python3-full for simplicity. IT IS bundled on python3-modules. > > I suggest one of the two solutions: > > 1. Accept python3-full and let it provide python3 > 2. Accept python3-full and let python3-core still provide python3. > > I would argue solution 1. is the more correct one, however it would increase > the size of artifacts for users who depend on python3 when they in fact > should > depend on python3-core, which I think is part of your argument. > > Solution 2. would still solve the problem. It would have been very clear to > me > and users what I need to get python3 and its entire library. > > Thus, if I understand your point correctly you are against solution 1. which > makes sense, however I don't see any reason for then not going with solution > 2. > > [1]: > http://lists.openembedded.org/pipermail/openembedded-core/2017-August/140830. > html Again, I dont believe this is a good solution, I dont think the patch is wrong I simply believe that the solution already exists, in any case, a discussion like this should be on the ML and not on the bug, so the whole community may comment (again, we shouldnt do what I say, we should do what is agreed by most people to be reasonable). If you really really really want to simply install python3-full just because you like the name better, a line like the following would probably do what you want and make you happy: PROVIDES_python3-modules += "python3-full" Anything else please lets discuss it on the ML.
I think folks may be talking slightly at cross-purposes here. FWIW, in master python3 *is* python-modules, so this should already work. See: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=62b28136c0114e4e9c19d3b62714bae23d73bc81 Linus, does this resolve the issue for you?
There you go. Thanks Paul!
Based on the previous discussion, python-modules can be considered as an existing answer for this request so I'm closing this now.
> I think folks may be talking slightly at cross-purposes here. FWIW, in master python3 *is* python-modules, so this should already work. See: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=62b28136c0114e4e9c19d3b62714bae23d73bc81 > > Linus, does this resolve the issue for you? Yes indeed, this is the fix for this issue :)
Closing as resolve fixed