Bug 14638

Summary: Implement new python packaging (PEP-517); egg-info => wheels
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Tim Orling <tim.orling>
Component: devtools / tool chainAssignee: Tim Orling <tim.orling>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: alex.kanavin, kweihmann, meta.mr.watcher, meta.watcher, randy.macleod, TicoTimo, tvgamblin
Version: 4.0   
Target Milestone: 4.0 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Bug Depends on:    
Bug Blocks: 14735    

Description Tim Orling 2021-11-25 04:34:01 UTC 
Upstream Python has been changing how packages are built, deployed and installed for quite some time now. The key PEP is 517:

https://www.python.org/dev/peps/pep-0517/

The crux of this change involves many related changes:

(1) setup.py will be replaced by pyproject.toml (and perhaps also setup.cfg)
(2) egg-info/eggs will be replaced by wheels
(3) packages are installed by pip, period.

For us, this means we need to use pip to install wheels (that we build) without dependencies into the sysroot. We will never allow pip to fetch wheels or other dependencies from the Internet. The wheels we build and then install into the sysroot (without dependencies) will be what is packaged for ipk/deb/rpm. We will NOT support installing third-party wheels in Yocto Project/OpenEmbedded recipes. The possible ways this can go wrong are infinite.

While we might hope to have a conversation with upstream and convince them to change their approach, the reality is this ship has sailed. We must adapt.

https://lists.openembedded.org/g/openembedded-architecture/message/1360

Working branch:
https://git.yoctoproject.org/poky-contrib/log/?h=timo/pypa-packaging
Comment 1 kweihmann 2021-11-25 08:54:53 UTC 
Judging from the upstream discussion https://github.com/pypa/setuptools/issues/1160#issuecomment-549054957 I'd say we definitely need to support both setup.cfg and *.toml
Comment 2 Alexander Kanavin 2021-12-06 21:28:08 UTC 
For what it's worth, there's now a recipe in oe-core with the issue (python3-jsonschema) - can't be updated because setup.py has vanished.
Comment 3 Tim Orling 2022-01-18 20:53:42 UTC 
flit_core 3.6.0 vendors tomli to get around the circular dependency issue (until some future date when a TOML parser is in the standard library).

https://github.com/pypa/flit/commit/55053217f09e052703c49b8812e7fa1a1a8359f4
Comment 4 Tim Orling 2022-02-22 20:34:00 UTC 
V3 of patch series:
https://patchwork.yoctoproject.org/project/oe-core/list/?series=1968
Comment 5 Tim Orling 2022-02-25 20:57:01 UTC 
This has been merged.

Any issues in oe-core should be either sent to the mailing list or a new bug filed.

Any issues with other layers should be sent to the appropriate mailing list.