pcp is part of meta-oe layer and when building it on Autobuilders occasionally a build failure is seen when building pcp-native recipe, it fails to link and errors are like below srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_sigmask@GLIBC_2.32' /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_condattr_setclock@GLIBC_2.34' /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_create@GLIBC_2.34' /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_join@GLIBC_2.34' collect2: error: ld returned 1 exit status This error is taken from a failed build on debian11 builder liblzma.so is coming from xz-native here and from sstate object which was built on a host that has newer glibc ( > 2.34 ) because the pthread_sigmask changed its API in glibc 2.32 due to pthread_attr_setsigmask_np and pthread_attr_getsigmask_np have been added. Which in turn cascades down to pthread_sigmask causing the symbol addition So with glibc >= 2.32 we get pthread_sigmask@GLIBC_2.32 and with glibc < 2.32 we get pthread_sigmask@GLIBC_2.2.5 see - https://sourceware.org/pipermail/libc-announce/2020/000029.html for the above mentioned changes. When xz-native is build on a host with glibc >= 2.32 we get first symbol linked in into liblzma and when xz-native is built on hosts with glibc < 2.32 it gets the second versioning symbol, however, when the library is brought in from sstate we do not know this distinction and if we do not use buildtools-tarball which has new enough glibc ( 2.34 + ) it will fail to link as seen above. I checked the autobuilder nodes and we have following nodes with glibc < 2.34 alma8 - glibc 2.28 debian11 - glibc 2.31 opensuse15.5 - glibc 2.31 rocky8 - glibc 2.28 ubuntu2004 - glibc 2.31 Which means it can cause the issue if the xz-native sstate artifact is built on builders running newer glibc and then used on one of above nodes from state. I would suggest that we use buildtools tarball on all nodes for consistency, if not all atleast on all nodes running glibc < 2.34, if we are already doing so then perhaps this analysis is not correct and I will explore other angles. Can someone take a look and confirm ?
I think this happens if you're building something that doesn't use BUILD_LDFLAGS. You can tell if the -Wl,--dynamic-linker and -Wl,--allowed-shlib-undefined options are being passed in. We shouldn't need to use buildtools.
Shlib-undefined option will defer the issue to runtime since the missing symbol is to be provided by libpthread which will come from host and ldso will complain when launching the native program on use
(In reply to Khem Raj from comment #2) > Shlib-undefined option will defer the issue to runtime since the missing > symbol is to be provided by libpthread which will come from host and ldso > will complain when launching the native program on use No, it won't. In a uninative build, a newer ld.so and libc.so are present so the symbols will be found, we point at these using the dynamic-linker option so the binary is linked to our own loader. This mechanism is used for all the binaries we build, it just requires BUILD_LDFLAGS to be used.
(In reply to Richard Purdie from comment #3) > (In reply to Khem Raj from comment #2) > > Shlib-undefined option will defer the issue to runtime since the missing > > symbol is to be provided by libpthread which will come from host and ldso > > will complain when launching the native program on use > > No, it won't. In a uninative build, a newer ld.so and libc.so are present so > the symbols will be found, we point at these using the dynamic-linker option > so the binary is linked to our own loader. This mechanism is used for all > the binaries we build, it just requires BUILD_LDFLAGS to be used. I checked the logs of pcp-native build locally and its using it correctly -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/mnt/b/yoe/master/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 please note these options will instruct it to ignore undefined symbols during static linking phase. Secondly it will encode the INTERP correctly, which I hope will be edited to point to new installation path when it is reused from sstate on different host. Now, I am not sure if ld.so from uninative will load the libc.so and libpthread.so from uninative as well. Sadly, the failed build is wiped out from the worker (debian11-vk-3) so I am not able to peek into it. This has to re-occur sometime again
(In reply to Khem Raj from comment #4) > (In reply to Richard Purdie from comment #3) > > (In reply to Khem Raj from comment #2) > > > Shlib-undefined option will defer the issue to runtime since the missing > > > symbol is to be provided by libpthread which will come from host and ldso > > > will complain when launching the native program on use > > > > No, it won't. In a uninative build, a newer ld.so and libc.so are present so > > the symbols will be found, we point at these using the dynamic-linker option > > so the binary is linked to our own loader. This mechanism is used for all > > the binaries we build, it just requires BUILD_LDFLAGS to be used. > > I checked the logs of pcp-native build locally and its using it correctly > > -Wl,--allow-shlib-undefined > -Wl,--dynamic-linker=/mnt/b/yoe/master/build/tmp/sysroots-uninative/x86_64- > linux/lib/ld-linux-x86-64.so.2 > > please note these options will instruct it to ignore undefined symbols > during static linking phase. Secondly it will encode the INTERP correctly, > which I hope will be edited to point to new installation path when it is > reused from sstate on different host. Now, I am not sure if ld.so from > uninative will load the libc.so and libpthread.so from uninative as well. > Sadly, the failed build is wiped out from the worker (debian11-vk-3) so I am > not able to peek into it. This has to re-occur sometime again another thing to note is that these symbols moved from libpthread.so into libc.so in glibc 2.32+,
Randy will find someone to work on this.
I did the test on Ubuntu 20.04.6 with glibc 2.31. $ cat /etc/issue Ubuntu 20.04.6 LTS \n \l $ ldd --version ldd (Ubuntu GLIBC 2.31-0ubuntu9.17) 2.31 Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper. And use the sstate cache which generated on Ubuntu 22.04.4(whith glibc 2.35), but didn't reproduce the issue until now.
This bug recently happened on AB (debian11-vk-1 worker): https://autobuilder.yoctoproject.org/valkyrie/#/builders/87/builds/51/steps/32/logs/stdio Full log: https://valkyrie.yocto.io/pub/repro-fail-openembedded/meta-oe/oe-reproducible-20250614-zcb89bjg/bitbake-reproducibleB-extended.log
Mingli is there a patch you can add so that if / when the problem happens again, we learn more about the bug?
I stil can't reproduce the issue locally and from the latest reproduce log https://errors.yoctoproject.org/Errors/Details/859030/, it seems it uses uninative linker as below: -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 The full log as below: 2025-06-14 06:48:48,442 - oe-selftest - INFO - | gcc -isystem/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/include -O2 -pipe -I./src/include -I./src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -Wall -O2 -g -I../src/include -I../src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -Wall -O2 -g -I../../src/include -I../../src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -DHAVE_TRACE -I../../src/pmdas/nvidia -Wall -O2 -g -o newcontext newcontext.c -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L./src/libpcp/src -L./src/libpcp_web/src -L./src/libpcp_pmda/src -L./src/libpcp_mmv/src -L./src/libpcp_archive/src -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L../src/libpcp/src -L../src/libpcp_web/src -L../src/libpcp_pmda/src -L../src/libpcp_mmv/src -L../src/libpcp_archive/src -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L../../src/libpcp/src -L../../src/libpcp_web/src -L../../src/libpcp_pmda/src -L../../src/libpcp_mmv/src -L../../src/libpcp_archive/src -L../../src/libpcp_gui/src -L../../src/libpcp_mmv/src -L../../src/libpcp_web/src -L../../src/libpcp_trace/src -L../../src/libpcp_import/src -lpcp 2025-06-14 06:48:48,442 - oe-selftest - INFO - | collect2: error: ld returned 1 exit status And then it prints the linker as "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld" | /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_join@GLIBC_2.34' 2025-06-14 08:14:38,494 - oe-selftest - INFO - | collect2: error: ld returned 1 exit status Not sure why the linker is "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld" though we pass "-Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2". Maybe the file /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 is missing when build pcp-native as some job is cleaning the file /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2?
Mingli, I think this is another bug due to sharing of sstate-cache between distros. What two linux distros and detailed steps have you taken to try to reproduce the error ?
Khem, Mingli, Is this still happening? Do you have more details ?
No info on whether this is still a problem, reopen if it is.
(In reply to Randy MacLeod from comment #13) > No info on whether this is still a problem, reopen if it is. It still is a problem, got it on AB today. https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/7464885/raw_inline
Hi Khem, Are we sure the recent failure is the same as the ones previously seen in 2025? The error seen in 2025 was "collect2: error: ld returned 1 exit status", with undefined references at link time. This looks like a potential issue with sharing sstate across distros. The recent build you link to shows two errors: - pcp-native fails with "fatal error: readline/readline.h: No such file or directory" - frr fails with clippy unable to load its shared libraries: "error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory" These look like missing dependencies or race conditions during the build. Perhaps we need to close this and open a new bug?
(In reply to Paul Barker from comment #15) > Hi Khem, > > Are we sure the recent failure is the same as the ones previously seen in > 2025? > > The error seen in 2025 was "collect2: error: ld returned 1 exit status", > with undefined references at link time. This looks like a potential issue > with sharing sstate across distros. > > The recent build you link to shows two errors: > - pcp-native fails with "fatal error: readline/readline.h: No such file or > directory" > - frr fails with clippy unable to load its shared libraries: "error while > loading shared libraries: libatomic.so.1: cannot open shared object file: No > such file or directory" > > These look like missing dependencies or race conditions during the build. > > Perhaps we need to close this and open a new bug? It's same problem now showing with different library. Symptoms is same where a native library added dependency based on host it was built on and sstate brought them over to the host that does not have the library