After the recent upgrade to python3-numpy 2.1.3, it was noticed that builds without pkgconfig (which the upgrade patch says was added, but wasn't) will fail on the do_compile step for certain hosts (e.g. F41). Adding pkgconfig to the inherit list for python3-numpy seems to fix this issue, but doesn't explain why it happens. Following some investigation, the following line was noticed in tmp/work/core2-64-poky-linux-musl/python3-numpy/2.1.3/build/meson-logs/meson-log.txt: |x86_64-poky-linux-musl-g++ -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/media/build/poky/build/tmp/work/core2-64-poky-linux-musl/python3-numpy/2.1.3/recipe-sysroot -I/media/build/poky/build/tmp/work/core2-64-poky-linux-musl/python3-numpy/2.1.3/recipe-sysroot/usr/include/python3.13 -I/usr/include/python3.13 ... The issue here being "-I/usr/include/python3.13", which should raise an error due to https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/gcc/gcc/0002-gcc-poison-system-directories.patch. It does so on F41, but on other systems (e.g. Debian 12), the build would finish without error. It turns out that this was because the poison patch only works correctly if /usr/include/python3.13 actually exists - which it doesn't on Debian 12 (and probably a lot of other systems). For numpy, we can get around this by adding the missing "inherit pkgconfig" to the recipe, but this should be fixed in gcc as well (and soon).
Sundeep, I won't have time to look at this after all. Can you do some investigation and submit a patch?
Hello Trevor, I tried reproducing the issue with 'bitbake python3-numpy' by adding the meta-musl & meta-clang layers in poky (along with required conf changes) to fix below errors. --------- WARNING: Layer musl-layer should set LAYERSERIES_COMPAT_musl-layer in its conf/layer.conf file to list the core layer names it is compatible with. ERROR: Variable BB_DANGLINGAPPENDS_WARNONLY support for the variable was removed as it unconditionally affects all layers (file: .../poky/meta-musl/conf/layer.conf line: 10) ERROR: Parse failure with the specified layer added, exiting. But still, I could see the error below: ERROR: No recipes in default available for: .../poky/meta-musl/recipes-core/llvm/llvm3.3_%.bbappend .../poky/meta-musl/recipes-core/systemd/systemd_225.bbappend ---------- Can you let us know the steps to reproduce the issue.
After further investigation, it has been determined that the meta-musl layer is not required. Instead, specifying TCLIB="musl" is sufficient for the desired configuration. I have successfully reproduced the issue on Fedora 41 and will continue debugging the problem to identify the root cause.
Summary of Issues Identified in the Numpy 2.1.3 Build Process: ---- Yocto utilizes the Meson build system to compile the Numpy package. Meson is closely integrated with the pkg-config tool, which is essential for managing and discovering project dependencies during the build process. Without pkg-config tool, two issues have been identified: Issue 1: Incorrect Inclusion of Host System Path In the absence of pkg-config, Meson defaults to its internal mechanism for locating Python header files. Meson correctly identifies the sysroot path (For cross compilation case). However, an issue arises when Meson incorrectly includes the host system path (/usr/include/python3.13), as seen in the following log snippet: Meson logs snippet: x86_64-poky-linux-g++ -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=...../tmp/work/core2-64-poky-linux/python3-numpy/2.1.3/recipe-sysroot -I ../build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.3/recipe-sysroot/usr/include/python3.13 -I/usr/include/python3.13 ----- When Python 3.13 is installed on the host system, the GCC poison mechanism triggers a warning, which is the expected behavior. However, when the pkg-config tool is absent, Meson including the host system path is a known issue, which persists even in the Meson master branch. This issue has been reported in the following Meson GitHub issue: https://github.com/mesonbuild/meson/issues/12540 While this issue does not block the build process when pkg-config is present. I have also reached out to the Meson project via IRC to seek clarification. It is likely that Meson’s cross-compilation support needs improvement to address this issue. Issue 2: Host System Path Inclusion when Python 3.13 is Not Installed If Python 3.13 is not installed on the host system, Meson still includes the host system path (/usr/include/python3.13) along with the sysroot path. In this case, the build will succeed since the Python header files are available in the sysroot path. However, GCC's poison mechanism does not issue a warning, which raises concerns about the expected behavior. I am currently investigating the following possibilities: 1) By design, GCC should issue a warning if the directory python3.13 exists on the host system. 2) Regardless of whether the directory exists, GCC should warn if anything under /usr/include is being searched. I am reviewing the GCC code to understand how it populates and verifies include paths.
Created attachment 5092 [details] Prevent host inclusion path during cross compilation Reproduction Steps: ****************** 1) Remove pkg-config from the numpy recipe (python3-numpy_2.1.3.bb). 2) Execute the command bitbake python3-numpy. Note: This issue is observed on any Linux-based distribution where Python 3.13 is not installed. Observed Behavior: Without the attached patch: The build process passes, though this is not the expected behavior. The build system silently ignores inclusion of host paths during cross-compilation. With the attached patch: GCC issues a warning about the include paths. Issue Analysis: GCC maintains various internal lists for include paths, such as INC_QUOTA for "file.h", and INC_BRACKET for <file.h>. In its current state, GCC silently ignores include paths that do not exist. This behavior is problematic as it can lead to unnoticed issues in cross-compilation scenarios. Fix Implementation: The patch modifies this behavior to ensure that missing include directories are preserved for cross-compilation scenarios. Specifically, the modification ensures that non-existing directories are retained in the include paths instead of being silently ignored. Expected Behavior: If the build system incorrectly adds a non-existent host directory path during a cross-compilation scenario, GCC should detect this and issue a warning.
If Meson were to use the -Wmissing-include-dirs flag in conjunction with -Werror for cross-compilation, it would eliminate the need for this additional check within GCC. I will post this on the meson's IRC.
I have submitted a modified GCC poison patch to the upstream, which can be found here: https://lists.openembedded.org/g/openembedded-core/message/209940 By integrating this check into GCC, it ensures that the validation applies across all packages and the build system. This helps identify if any part of the build system attempts to search the host system's path during cross-compilation. Associated logs: Host Python Version: ------------------- python3 --version Python 3.10.12 With the application of this GCC check, I observed that GCC prevents the inclusion of a non-existent directory (in this case, python3.13) when building the Numpy package. x86_64-poky-linux-g++ -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/build/tmp/work/core2-64-poky-linux/python3-numpy/2.2.1/recipe-sysroot -I /build/tmp/work/core2-64-poky-linux/python3-numpy/2.2.1/recipe-sysroot/usr/include/python3.13 -I/usr/include/python3.13 ----- cc1plus: error: include location "/usr/include/python3.13" is unsafe for cross-compilation [-Werror=poison-system-directories] cc1plus: some warnings being treated as errors
The patch was merged in oe-core 32e2f9a73a653a00e555e50f9197b605f2f70f89.