Bug 15312 - python3-maturin: Add 'maturin' PEP-517 backend support
Summary: python3-maturin: Add 'maturin' PEP-517 backend support
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 5.0
Hardware: All Multiple
: Medium+ major
Target Milestone: 5.0 M2
Assignee: Tim Orling
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2023-12-07 22:40 UTC by Tim Orling
Modified: 2023-12-17 20:15 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Orling 2023-12-07 22:40:04 UTC
We are increasingly seeing Rust extensions for Python added in the
ecosystem. The current best practices approach has moved from using
'setuptools-rust' to using 'maturin' [1].

We absolutely need to make sure this makes it into the future LTS release (5.0 "scarthgap").

* python3-pydantic now requires python3-pydantic-core which is built with 'maturin' [2]
* python3-orjson [3] is now commonly being used for JSON support
* python3-ruff [4] is a (very fast) Python linter and is increasingly being used

[1] https://pypi.org/project/maturin/
[2] https://pypi.org/project/pydantic-core/
[3] https://pypi.org/project/orjson/
[4] https://pypi.org/project/ruff/
Comment 3 Tim Orling 2023-12-07 23:32:47 UTC
An additional (much deeper) functional test is at the tip of:
https://git.yoctoproject.org/poky-contrib/log/?h=timo/maturin_wip

https://git.yoctoproject.org/poky-contrib/commit/?h=timo/maturin_wip&id=395521b21afc2be8298ab16ca2c2ad3a75c28864

But that is currently failing for Rust _reasons_, not related to the
test case most likely. Put that test case on a back burner for now.
Comment 4 Pascal Bach 2023-12-12 08:16:55 UTC
I was able to build https://github.com/eclipse-zenoh/zenoh-python using the https://git.yoctoproject.org/poky-contrib/log/?h=timo/maturin_1.4.0 branch.

I had to run `bitbake python3-maturin-native -c update_crates` as the crate list seems to have been out of date. Everything else worked.
Comment 5 Tim Orling 2023-12-12 18:36:00 UTC
@Pascal:
Thank you for reminding me of the "obvious". I was just banging my head on the fetch failure...
Comment 6 Tim Orling 2023-12-17 20:15:47 UTC
https://lore.kernel.org/openembedded-core/cover.1702791173.git.tim.orling@konsulko.com/T/#t

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6341
(failures were completely unrelated to maturin)

Merged as of commit:
https://git.yoctoproject.org/poky/commit/?id=7d881f0214f9761990e52595d33246700c4da15c


From the prior series, the main issue was the bzip2-sys crate needed a debug path mapping from CARGO_HOME (used TARGET_DBGSRC_DIR/cargo_home/ as the mapped path) in order to not inject TMPDIR into DW_AT_dir in /usr/bin/.debug/maturin (which also had the side effect of causing reordering). This resulted in a non-reproducible build.

The solution that actually worked was to set the path mapping in CFLAGS (both a patch to build.rs in bzip2-sys and higher level RUST_DEBUG_REMAP were also attempted).
CFLAGS:append = "-fdebug-prefix-map=${CARGO_HOME}=${TARGET_DBGSRC_DIR}/cargo_home"

The second issue was figuring out how to get the 'maturin develop' testsdk case to not be skipped, which requires a full Rust environment in the SDK. This turned out to be two things in addition to TOOLCHAIN_HOST_TASK (which are set on the AB):
TOOLCHAIN_HOST_TASK:append = " nativesdk-python3-maturin"
SDK_INCLUDE_TOOLCHAIN = '1'
SDK_TOOLCHAIN_LANGS += 'rust'

testimage and testsdk were tested on qemux86-64 and qemuarm64.
rust.RustCompileTest.test_cargo_build and
maturin.MaturinDevelopTest.test_maturin_develop failed on qemuarm64 (SDKMACHINE x86_64 and i686).