Bug 15579

The project https://github.com/aabelix/rust-paho-mqtt-example builds fine with SDK compiled from nanbield

DISTRO = "poky"
DISTRO_NAME = "Poky (Yocto Project Reference Distro)"
DISTRO_VERSION = "4.3.4"
DISTRO_CODENAME = "nanbield"

and that has glibc 2.38.

Yash saw a similiar problem when updating to rust-1.78.
Nanbield isn't supported but we can share what we had to do to resolve the issue and Aapo can try that.

> Yash saw a similiar problem when updating to rust-1.78. Nanbield isn't supported but we can share what we had to do to resolve the issue and Aapo can try that.

I am not sure if I follow. I want to clarify that we are trying to upgrade from kirkstone to scarthgap thats when I found about this issue. I just ran a test on previous release branch (=nanbield) to see if this issue affects scarthgap and it seems so because I am not able to reproduce this on nanbield.

Nanbield is using rust 1.75 rust and scarthgap has 1.78 so it could be caused by the rust version upgrade in scarthgap. Shouldn't this issue be fixed for scarthgap which is a LTS release?

If there are patch sets that fix this, could I be pointed to those patch-sets so I can try to port them to our BSP?

Aapo, Thanks for the clarification.

Your branch/version of rust seems to be different from the main version.

Are you using a custom set of changes on top of oe-core/poky?
Are you using the meta-rust layer in addition to the rust support in oe-core?
If so, this would technically NOT be a bug for oe-core.

I see:

poky.git on nanbield
❯ fd rust_
..
meta/recipes-devtools/rust/rust_1.70.0.bb

poky.git on scarthgap
❯ fd rust_
..
meta/recipes-devtools/rust/rust_1.75.0.bb




Yash's comments are in this oe-core thread:

https://lore.kernel.org/openembedded-core/20240821104824.6IeRC1JkTuOfDwpx5nEVavdq2owUYGzWNPYLLSDJ9mI@z/

Subject: rust: Oe-selftest changes for rust v1.78

As Randy mentioned,
Nanbield has rust v1.70 and scarthgap has v1.75 and it's not updated to v1.78 yet.
Please let us know if you are making any custom changes on top of oe-core/poky.

Oe-core/master was upgraded with v1.78 recently. 
If you wish to update to v1.78, please apply the following commit logs and let us know the results there:
https://git.yoctoproject.org/poky/log/meta/recipes-devtools/rust?h=master

> Your branch/version of rust seems to be different from the main version.

Yeah sorry for my miscommunication. Our custom BSP has rust 1.78 but poky scarthgap has 1.75.

These are the literal instructions that I used to build SDK:

$ git clone --branch scarthgap git://git.yoctoproject.org/poky
$ cd poky
poky$ source oe-init-build-env
poky/build$ echo 'SDK_TOOLCHAIN_LANGS += "rust"' >> conf/local.conf
poky/build$ bitbake core-image-minimal -c populate_sdk -k

(same stepa are repeated here: https://github.com/aabelix/rust-paho-mqtt-example)

I am also seeing this same issue on our custom BSP layer that uses rust 1.78 but it seem to affect poky scarthgap, too.

Yash, please try the reproducer and report back here. Thanks.

(In reply to Aapo Naalisvaara from comment #0)
> After populating SDK with Rust language support, the SDK fails to compile
> Rust module `paho-mqtt` due to
> 
> lib/libc.so.6: undefined symbol: __tunable_is_initialized, version
> GLIBC_PRIVATE 
> 
> I have built the SDK with the following 
> 
> $ source oe-init-build-env
> build$ echo 'SDK_TOOLCHAIN_LANGS += "rust"' >> conf/local.conf
> build$ bitbake core-image-minimal -c populate_sdk
> 
> To help reproducing the issue, I have made a sample github project:
> https://github.com/aabelix/rust-paho-mqtt-example.
> 
> I am able to build rust packages that depend on the `paho-mqtt` module on
> our own BSP with Yocto Scarthgap so to me this seems related to the build
> configuration of nativesdk of glibc.

what is the build host OS and version you are running on ?

I think the problem is that its needing libc from nativesdk-glibc
however its being redirected to the one on the SDK host after SDK is installed.

Can you try installing extended buildtools tarball and build the SDK in that env.
you can install it with scripts/install-buildtools tool

(In reply to Randy MacLeod from comment #7)
> Yash, please try the reproducer and report back here. Thanks.

The reproducer worked and I see the error.


(In reply to Khem Raj from comment #8)
> 
> what is the build host OS and version you are running on ?
> 
> I think the problem is that its needing libc from nativesdk-glibc
> however its being redirected to the one on the SDK host after SDK is
> installed.
> 
> Can you try installing extended buildtools tarball and build the SDK in that
> env.
> you can install it with scripts/install-buildtools tool
I checked with installing the buildtools tarball and build the SDK from it.
The error is seen again.

Recently, there was some linking error seen with 1.78 on AB for alma9 distro.
The discussion thread can be found here:
https://lists.openembedded.org/g/openembedded-core/message/203827

I enabled "-fPIE" for zlib recipe(as mentioned in the discussion), checked the reproducer and still the  issue exists.

> I am not sure if I follow. I want to clarify that we are trying to upgrade
> from kirkstone to scarthgap thats when I found about this issue. I just ran
> a test on previous release branch (=nanbield) to see if this issue affects
> scarthgap and it seems so because I am not able to reproduce this on
> nanbield.


I am able to reproduce this issue on nanbield(rust v1.70) as well.
Can you please check and confirm the same?

Please let me know with which rust version or poky branch it worked fine.

The following observations were made:

1. The logs indicate that the SDK-generated GCC compiler appears to be malfunctioning. The check for a functional C compiler at /yocto/poky/build/y/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc has failed, as even a simple hello.c program does not compile with it.

2. The cargo.lock file contains a list of dependent crates. The "cc" crate included in this list relies on the SDK-generated C compiler to compile and link the crate being built. Consequently, an error occurs when attempting to build the paho-mqtt crate.

3. In contrast, other crates build successfully without any errors in the SDK environment using the SDK-generated cargo binary. This suggests that there may be a specific issue with building this particular crate.

4. Additionally, I checked other branches (including nanbield, scarthgap, and master) and confirmed with the reporter via email that the issue has never functioned in those branches. It appears that this problem has existed from the beginning.

I think I have found the root for this error. The SDK uses a wrapper around Cargo that sets the LD_LIBRARY_PATH enviroment variable to the SDK target library directory. This is causing build scripts and simple commands such as `cargo help build` to fail, because they are launching executables that need to link the host system.

As a workaround one can use `cargo.real` instead of `cargo`. Could you check if using `cargo.real` is solving the problem?

I'm about to submit a patch that is completly removing the wrapper but need to run the self tests first. And building a new image with rust enabled takes a long time...

(In reply to Jan Strater-Büddefeld from comment #12)
> I think I have found the root for this error. The SDK uses a wrapper around
> Cargo that sets the LD_LIBRARY_PATH enviroment variable to the SDK target
> library directory. This is causing build scripts and simple commands such as
> `cargo help build` to fail, because they are launching executables that need
> to link the host system.
> 
> As a workaround one can use `cargo.real` instead of `cargo`. Could you check
> if using `cargo.real` is solving the problem?
Yes, cargo.real does work. I was checking the diff between 'cargo' and 'cargo.real' 
and it seems 'cargo' is a bash script whereas 'cargo.real' is a ELF pie executable.

> 
> I'm about to submit a patch that is completly removing the wrapper but need
There's a wrapper in cargo recipe as follows in:
"create_wrapper ${D}/${bindir}/cargo LD_LIBRARY_PATH=${libdir}:${base_libdir}"

The above change was added to work around host system library conflicts(maybe for tumbleweed-ty-3).
https://git.openembedded.org/openembedded-core/commit/?id=388e7cac9f90e79ce8c3c1683d8ee0f4df1bc907 
It can be skipped if that's not an issue with newer rust versions and doesn't cause other failures.
Let me know if we are on the same page or if you have any other changes/suggestions.

> to run the self tests first. And building a new image with rust enabled
> takes a long time...
Yeah, it does take a long time to build and test rust.
Meanwhile, I will also perform some tests without the wrapper.

> The above change was added to work around host system library conflicts(maybe for tumbleweed-ty-3).
> https://git.openembedded.org/openembedded-core/commit/?id=388e7cac9f90e79ce8c3c1683d8ee0f4df1bc907 
> It can be skipped if that's not an issue with newer rust versions and doesn't cause other failures.
> Let me know if we are on the same page or if you have any other changes/suggestions.

Yes correctly. Unfortunately cargo still doesn't set LD_LIBRARY_PATH to /lib so removing the wrapper will probably cause failures on this machines again. But I would argue that the current wrapper causes more harm than it solves and one can always set LD_LIBRARY_PATH manually if needed.

I am not aware if Yocto still use tumbleweed for builds and testings.

If not, can we drop the wrapper in 
https://git.openembedded.org/openembedded-core/commit/?id=388e7cac9f90e79ce8c3c1683d8ee0f4df1bc907 ?

I guess Richard can provide more information on this.
CC: Richard

It looks like tumbleweed is not in the list of tested distros:

https://git.yoctoproject.org/yocto-autobuilder2/tree/config.py?id=694d2a9bae523d9396b37da9cc6535a558e04d81#n168

It was there previously:
https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=96e82ce670c02b166398500435c2df455b09b951

Richard?

From IRC:
[10:38] <RP> vmeson: yes, it was the hardest to maintain distro and is "floating" so we can't document a version either. I gave "permission" to drop it

I have submitted a patch to the mailing list that removes the wrapper.

Thank you for sharing the correct project tree for reproducing the issue.
The issue is reproduced now.

Could you please use "cargo.real build" command to build the project and let us know the results?
For us, we didn't encounter the above errors but it failed while creating a directory during one of the cmake commands as follows (maybe since we don't have root access on our working machine):

  --- stderr
  ---CMAKE_HOST_SYSTEM_PROCESSOR: unknown
  ---CMAKE_SYSTEM_PROCESSOR: aarch64
  CMake Error at cmake_install.cmake:60 (file):
    file cannot create directory: /home/adi.  Maybe need administrative
    privileges.

  gmake: *** [Makefile:103: install] Error 1
  thread 'main' panicked at /home/sdk/sysroots/core2-64-poky-linux/home/cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.53/src/lib.rs:1121:5:
  command did not execute successfully, got: exit status: 2

It tries to create a dir at home path via following cmd in cmake_install.cmake:60  :

file(INSTALL DESTINATION "/home/adi" TYPE EXECUTABLE FILES "/home/user_abc/15735/test_cmake/target/aarch64-poky-linux-gnu/debug/build/test_cmake-92b80e12c468f880/out/build/tutorial")

It seems this issue is similar to one reported in  https://bugzilla.yoctoproject.org/show_bug.cgi?id=15579.
The cargo build error is seen because of the wrapper present in cargo recipe.
I believe it should work fine without the wrapper.

The above comment was meant to be for https://bugzilla.yoctoproject.org/show_bug.cgi?id=15735.
(similar issue as this one)

It was added here by mistake.

The root cause of this SDK failure was identified as the use of a wrapper in the cargo recipe:

https://git.openembedded.org/openembedded-core/commit/?id=388e7cac9f90e79ce8c3c1683d8ee0f4df1bc907
https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/rust/cargo_1.89.0.bb#n51


This wrapper was introduced to address conflicts with host system libraries, specifically 
in Tumbleweed-based systems (e.g., Tumbleweed-ty-3), by appending /lib/ to LD_LIBRARY_PATH 
during SDK builds.
However, Tumbleweed is no longer part of the officially tested distributions in the 
Yocto AB due to maintenance challenges:

https://git.yoctoproject.org/yocto-autobuilder2/tree/config.py?id=694d2a9bae523d9396b37da9cc6535a558e04d81#n168
https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=96e82ce670c02b166398500435c2df455b09b951


Complete build tests were performed with the wrapper removed and no regressions or issues were observed. 
A patch was proposed to drop the wrapper.
https://lists.openembedded.org/g/openembedded-core/message/210088
https://lists.openembedded.org/g/openembedded-core/message/210089

Although the wrapper is currently retained to mitigate potential issues with combinations of host and nativesdk glibc,
an alternative fix is to use the 'cargo.real' binary instead of the wrapped cargo when building projects within the SDK.

Mark as Won't fix.

*** Bug 15735 has been marked as a duplicate of this bug. ***