Bug 14638 - Implement new python packaging (PEP-517); egg-info => wheels
Summary: Implement new python packaging (PEP-517); egg-info => wheels
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 4.0
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 4.0 M3
Assignee: Tim Orling
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks: 14735
  Show dependency tree
 
Reported: 2021-11-25 04:34 UTC by Tim Orling
Modified: 2022-02-25 20:57 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Orling 2021-11-25 04:34:01 UTC
Upstream Python has been changing how packages are built, deployed and installed for quite some time now. The key PEP is 517:

https://www.python.org/dev/peps/pep-0517/

The crux of this change involves many related changes:

(1) setup.py will be replaced by pyproject.toml (and perhaps also setup.cfg)
(2) egg-info/eggs will be replaced by wheels
(3) packages are installed by pip, period.

For us, this means we need to use pip to install wheels (that we build) without dependencies into the sysroot. We will never allow pip to fetch wheels or other dependencies from the Internet. The wheels we build and then install into the sysroot (without dependencies) will be what is packaged for ipk/deb/rpm. We will NOT support installing third-party wheels in Yocto Project/OpenEmbedded recipes. The possible ways this can go wrong are infinite.

While we might hope to have a conversation with upstream and convince them to change their approach, the reality is this ship has sailed. We must adapt.

https://lists.openembedded.org/g/openembedded-architecture/message/1360

Working branch:
https://git.yoctoproject.org/poky-contrib/log/?h=timo/pypa-packaging
Comment 1 kweihmann 2021-11-25 08:54:53 UTC
Judging from the upstream discussion https://github.com/pypa/setuptools/issues/1160#issuecomment-549054957 I'd say we definitely need to support both setup.cfg and *.toml
Comment 2 Alexander Kanavin 2021-12-06 21:28:08 UTC
For what it's worth, there's now a recipe in oe-core with the issue (python3-jsonschema) - can't be updated because setup.py has vanished.
Comment 3 Tim Orling 2022-01-18 20:53:42 UTC
flit_core 3.6.0 vendors tomli to get around the circular dependency issue (until some future date when a TOML parser is in the standard library).

https://github.com/pypa/flit/commit/55053217f09e052703c49b8812e7fa1a1a8359f4
Comment 4 Tim Orling 2022-02-22 20:34:00 UTC
V3 of patch series:
https://patchwork.yoctoproject.org/project/oe-core/list/?series=1968
Comment 5 Tim Orling 2022-02-25 20:57:01 UTC
This has been merged.

Any issues in oe-core should be either sent to the mailing list or a new bug filed.

Any issues with other layers should be sent to the appropriate mailing list.