15769 2025-02-21 03:08:35 +0000 pcp-native intermittent build failure 2026-07-31 16:00:59 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple NEW AB-INT Medium+ normal 6.1 M2 1 raj.khem raj.khem hongxu.jia infras.ab.watcher Infras.watcher paul paulg pidge randy.macleod tim.orling yoann.congal Ubuntu --- Don't know oldest_to_newest 101195 0 raj.khem 2025-02-21 03:08:35 +0000 pcp is part of meta-oe layer and when building it on Autobuilders occasionally a build failure is seen when building pcp-native recipe, it fails to link and errors are like below srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_sigmask@GLIBC_2.32' /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_condattr_setclock@GLIBC_2.34' /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_create@GLIBC_2.34' /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_join@GLIBC_2.34' collect2: error: ld returned 1 exit status This error is taken from a failed build on debian11 builder liblzma.so is coming from xz-native here and from sstate object which was built on a host that has newer glibc ( > 2.34 ) because the pthread_sigmask changed its API in glibc 2.32 due to pthread_attr_setsigmask_np and pthread_attr_getsigmask_np have been added. Which in turn cascades down to pthread_sigmask causing the symbol addition So with glibc >= 2.32 we get pthread_sigmask@GLIBC_2.32 and with glibc < 2.32 we get pthread_sigmask@GLIBC_2.2.5 see - https://sourceware.org/pipermail/libc-announce/2020/000029.html for the above mentioned changes. When xz-native is build on a host with glibc >= 2.32 we get first symbol linked in into liblzma and when xz-native is built on hosts with glibc < 2.32 it gets the second versioning symbol, however, when the library is brought in from sstate we do not know this distinction and if we do not use buildtools-tarball which has new enough glibc ( 2.34 + ) it will fail to link as seen above. I checked the autobuilder nodes and we have following nodes with glibc < 2.34 alma8 - glibc 2.28 debian11 - glibc 2.31 opensuse15.5 - glibc 2.31 rocky8 - glibc 2.28 ubuntu2004 - glibc 2.31 Which means it can cause the issue if the xz-native sstate artifact is built on builders running newer glibc and then used on one of above nodes from state. I would suggest that we use buildtools tarball on all nodes for consistency, if not all atleast on all nodes running glibc < 2.34, if we are already doing so then perhaps this analysis is not correct and I will explore other angles. Can someone take a look and confirm ? 101201 1 richard.purdie 2025-02-21 13:02:09 +0000 I think this happens if you're building something that doesn't use BUILD_LDFLAGS. You can tell if the -Wl,--dynamic-linker and -Wl,--allowed-shlib-undefined options are being passed in. We shouldn't need to use buildtools. 101203 2 raj.khem 2025-02-21 15:11:14 +0000 Shlib-undefined option will defer the issue to runtime since the missing symbol is to be provided by libpthread which will come from host and ldso will complain when launching the native program on use 101209 3 richard.purdie 2025-02-24 10:48:56 +0000 (In reply to Khem Raj from comment #2) > Shlib-undefined option will defer the issue to runtime since the missing > symbol is to be provided by libpthread which will come from host and ldso > will complain when launching the native program on use No, it won't. In a uninative build, a newer ld.so and libc.so are present so the symbols will be found, we point at these using the dynamic-linker option so the binary is linked to our own loader. This mechanism is used for all the binaries we build, it just requires BUILD_LDFLAGS to be used. 101212 4 raj.khem 2025-02-25 02:35:02 +0000 (In reply to Richard Purdie from comment #3) > (In reply to Khem Raj from comment #2) > > Shlib-undefined option will defer the issue to runtime since the missing > > symbol is to be provided by libpthread which will come from host and ldso > > will complain when launching the native program on use > > No, it won't. In a uninative build, a newer ld.so and libc.so are present so > the symbols will be found, we point at these using the dynamic-linker option > so the binary is linked to our own loader. This mechanism is used for all > the binaries we build, it just requires BUILD_LDFLAGS to be used. I checked the logs of pcp-native build locally and its using it correctly -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/mnt/b/yoe/master/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 please note these options will instruct it to ignore undefined symbols during static linking phase. Secondly it will encode the INTERP correctly, which I hope will be edited to point to new installation path when it is reused from sstate on different host. Now, I am not sure if ld.so from uninative will load the libc.so and libpthread.so from uninative as well. Sadly, the failed build is wiped out from the worker (debian11-vk-3) so I am not able to peek into it. This has to re-occur sometime again 101215 5 raj.khem 2025-02-25 03:22:34 +0000 (In reply to Khem Raj from comment #4) > (In reply to Richard Purdie from comment #3) > > (In reply to Khem Raj from comment #2) > > > Shlib-undefined option will defer the issue to runtime since the missing > > > symbol is to be provided by libpthread which will come from host and ldso > > > will complain when launching the native program on use > > > > No, it won't. In a uninative build, a newer ld.so and libc.so are present so > > the symbols will be found, we point at these using the dynamic-linker option > > so the binary is linked to our own loader. This mechanism is used for all > > the binaries we build, it just requires BUILD_LDFLAGS to be used. > > I checked the logs of pcp-native build locally and its using it correctly > > -Wl,--allow-shlib-undefined > -Wl,--dynamic-linker=/mnt/b/yoe/master/build/tmp/sysroots-uninative/x86_64- > linux/lib/ld-linux-x86-64.so.2 > > please note these options will instruct it to ignore undefined symbols > during static linking phase. Secondly it will encode the INTERP correctly, > which I hope will be edited to point to new installation path when it is > reused from sstate on different host. Now, I am not sure if ld.so from > uninative will load the libc.so and libpthread.so from uninative as well. > Sadly, the failed build is wiped out from the worker (debian11-vk-3) so I am > not able to peek into it. This has to re-occur sometime again another thing to note is that these symbols moved from libpthread.so into libc.so in glibc 2.32+, 101241 6 randy.macleod 2025-02-27 15:37:45 +0000 Randy will find someone to work on this. 101711 7 mingli.yu 2025-04-23 08:54:52 +0000 I did the test on Ubuntu 20.04.6 with glibc 2.31. $ cat /etc/issue Ubuntu 20.04.6 LTS \n \l $ ldd --version ldd (Ubuntu GLIBC 2.31-0ubuntu9.17) 2.31 Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper. And use the sstate cache which generated on Ubuntu 22.04.4(whith glibc 2.35), but didn't reproduce the issue until now. 102320 8 yoann.congal 2025-06-21 15:23:04 +0000 This bug recently happened on AB (debian11-vk-1 worker): https://autobuilder.yoctoproject.org/valkyrie/#/builders/87/builds/51/steps/32/logs/stdio Full log: https://valkyrie.yocto.io/pub/repro-fail-openembedded/meta-oe/oe-reproducible-20250614-zcb89bjg/bitbake-reproducibleB-extended.log 102337 9 randy.macleod 2025-06-24 20:47:31 +0000 Mingli is there a patch you can add so that if / when the problem happens again, we learn more about the bug? 102418 10 mingli.yu 2025-07-04 10:22:22 +0000 I stil can't reproduce the issue locally and from the latest reproduce log https://errors.yoctoproject.org/Errors/Details/859030/, it seems it uses uninative linker as below: -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 The full log as below: 2025-06-14 06:48:48,442 - oe-selftest - INFO - | gcc -isystem/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/include -O2 -pipe -I./src/include -I./src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -Wall -O2 -g -I../src/include -I../src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -Wall -O2 -g -I../../src/include -I../../src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -DHAVE_TRACE -I../../src/pmdas/nvidia -Wall -O2 -g -o newcontext newcontext.c -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L./src/libpcp/src -L./src/libpcp_web/src -L./src/libpcp_pmda/src -L./src/libpcp_mmv/src -L./src/libpcp_archive/src -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L../src/libpcp/src -L../src/libpcp_web/src -L../src/libpcp_pmda/src -L../src/libpcp_mmv/src -L../src/libpcp_archive/src -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L../../src/libpcp/src -L../../src/libpcp_web/src -L../../src/libpcp_pmda/src -L../../src/libpcp_mmv/src -L../../src/libpcp_archive/src -L../../src/libpcp_gui/src -L../../src/libpcp_mmv/src -L../../src/libpcp_web/src -L../../src/libpcp_trace/src -L../../src/libpcp_import/src -lpcp 2025-06-14 06:48:48,442 - oe-selftest - INFO - | collect2: error: ld returned 1 exit status And then it prints the linker as "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld" | /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_join@GLIBC_2.34' 2025-06-14 08:14:38,494 - oe-selftest - INFO - | collect2: error: ld returned 1 exit status Not sure why the linker is "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld" though we pass "-Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2". Maybe the file /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 is missing when build pcp-native as some job is cleaning the file /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2? 103869 11 randy.macleod 2026-01-13 22:42:51 +0000 Mingli, I think this is another bug due to sharing of sstate-cache between distros. What two linux distros and detailed steps have you taken to try to reproduce the error ? 105659 12 randy.macleod 2026-05-28 15:25:11 +0000 Khem, Mingli, Is this still happening? Do you have more details ? 105830 13 randy.macleod 2026-06-11 15:00:36 +0000 No info on whether this is still a problem, reopen if it is. 106055 14 raj.khem 2026-07-01 06:44:09 +0000 (In reply to Randy MacLeod from comment #13) > No info on whether this is still a problem, reopen if it is. It still is a problem, got it on AB today. https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/7464885/raw_inline 106400 15 paul 2026-07-31 09:04:25 +0000 Hi Khem, Are we sure the recent failure is the same as the ones previously seen in 2025? The error seen in 2025 was "collect2: error: ld returned 1 exit status", with undefined references at link time. This looks like a potential issue with sharing sstate across distros. The recent build you link to shows two errors: - pcp-native fails with "fatal error: readline/readline.h: No such file or directory" - frr fails with clippy unable to load its shared libraries: "error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory" These look like missing dependencies or race conditions during the build. Perhaps we need to close this and open a new bug? 106409 16 raj.khem 2026-07-31 16:00:59 +0000 (In reply to Paul Barker from comment #15) > Hi Khem, > > Are we sure the recent failure is the same as the ones previously seen in > 2025? > > The error seen in 2025 was "collect2: error: ld returned 1 exit status", > with undefined references at link time. This looks like a potential issue > with sharing sstate across distros. > > The recent build you link to shows two errors: > - pcp-native fails with "fatal error: readline/readline.h: No such file or > directory" > - frr fails with clippy unable to load its shared libraries: "error while > loading shared libraries: libatomic.so.1: cannot open shared object file: No > such file or directory" > > These look like missing dependencies or race conditions during the build. > > Perhaps we need to close this and open a new bug? It's same problem now showing with different library. Symptoms is same where a native library added dependency based on host it was built on and sstate brought them over to the host that does not have the library