Bug 15769

Summary: pcp-native intermittent build failure
Product: [Infrastructure] AutoBuilder Reporter: Khem Raj <raj.khem>
Component: autobuilderAssignee: Khem Raj <raj.khem>
Status: NEW --- QA Contact:
Severity: normal    
Priority: Medium+ CC: hongxu.jia, infras.ab.watcher, Infras.watcher, paul, paulg, pidge, randy.macleod, tim.orling, yoann.congal
Version: unspecified   
Target Milestone: 6.1 M2   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: Ubuntu Type of Regression: ---
Verified: Documentation change: Don't know

Description Khem Raj 2025-02-21 03:08:35 UTC 
pcp is part of meta-oe layer and when building it on Autobuilders occasionally a build failure is seen when building pcp-native recipe, it fails to link and errors are like below

srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_sigmask@GLIBC_2.32'
/srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_condattr_setclock@GLIBC_2.34'
/srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_create@GLIBC_2.34'
/srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/x86_64-linux/pcp-native/6.0.5/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_join@GLIBC_2.34'
collect2: error: ld returned 1 exit status


This error is taken from a failed build on debian11 builder

liblzma.so is coming from xz-native here and from sstate object which was built on a host that has newer glibc ( > 2.34 ) because the pthread_sigmask changed its API in glibc 2.32 due to pthread_attr_setsigmask_np and
  pthread_attr_getsigmask_np have been added. Which in turn cascades down to pthread_sigmask causing the symbol addition

So with glibc >= 2.32 we get pthread_sigmask@GLIBC_2.32 and with glibc < 2.32 we get pthread_sigmask@GLIBC_2.2.5

see - https://sourceware.org/pipermail/libc-announce/2020/000029.html
for the above mentioned changes.

When xz-native is build on a host with glibc >= 2.32 we get first symbol linked in into liblzma and when xz-native is built on hosts with glibc < 2.32 it gets the second versioning symbol, however, when the library is brought in from sstate we do not know this distinction and if we do not use buildtools-tarball which has new enough glibc ( 2.34 + ) it will fail to link as seen above.

I checked the autobuilder nodes and we have following nodes with glibc < 2.34

alma8 - glibc 2.28
debian11 - glibc 2.31
opensuse15.5 - glibc 2.31
rocky8 - glibc 2.28
ubuntu2004 - glibc 2.31

Which means it can cause the issue if the xz-native sstate artifact is built on builders running newer glibc and then used on one of above nodes from state.

I would suggest that we use buildtools tarball on all nodes for consistency, if not all atleast on all nodes running glibc < 2.34, if we are already doing so then perhaps this analysis is not correct and I will explore other angles.

Can someone take a look and confirm ?
Comment 1 Richard Purdie 2025-02-21 13:02:09 UTC 
I think this happens if you're building something that doesn't use BUILD_LDFLAGS. You can tell if the -Wl,--dynamic-linker  and -Wl,--allowed-shlib-undefined options are being passed in. We shouldn't need to use buildtools.
Comment 2 Khem Raj 2025-02-21 15:11:14 UTC 
Shlib-undefined option will defer the issue to runtime since the missing symbol is to be provided by libpthread which will come from host and ldso will complain when launching the native program on use
Comment 3 Richard Purdie 2025-02-24 10:48:56 UTC 
(In reply to Khem Raj from comment #2)
> Shlib-undefined option will defer the issue to runtime since the missing
> symbol is to be provided by libpthread which will come from host and ldso
> will complain when launching the native program on use

No, it won't. In a uninative build, a newer ld.so and libc.so are present so the symbols will be found, we point at these using the dynamic-linker option so the binary is linked to our own loader. This mechanism is used for all the binaries we build, it just requires BUILD_LDFLAGS to be used.
Comment 4 Khem Raj 2025-02-25 02:35:02 UTC 
(In reply to Richard Purdie from comment #3)
> (In reply to Khem Raj from comment #2)
> > Shlib-undefined option will defer the issue to runtime since the missing
> > symbol is to be provided by libpthread which will come from host and ldso
> > will complain when launching the native program on use
> 
> No, it won't. In a uninative build, a newer ld.so and libc.so are present so
> the symbols will be found, we point at these using the dynamic-linker option
> so the binary is linked to our own loader. This mechanism is used for all
> the binaries we build, it just requires BUILD_LDFLAGS to be used.

I checked the logs of pcp-native build locally and its using it correctly

-Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/mnt/b/yoe/master/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2

please note these options will instruct it to ignore undefined symbols during static linking phase. Secondly it will encode the INTERP correctly, which I hope will be edited to point to new installation path when it is reused from sstate on different host. Now, I am not sure if ld.so from uninative will load the libc.so and libpthread.so from uninative as well. Sadly, the failed build is wiped out from the worker (debian11-vk-3) so I am not able to peek into it. This has to re-occur sometime again
Comment 5 Khem Raj 2025-02-25 03:22:34 UTC 
(In reply to Khem Raj from comment #4)
> (In reply to Richard Purdie from comment #3)
> > (In reply to Khem Raj from comment #2)
> > > Shlib-undefined option will defer the issue to runtime since the missing
> > > symbol is to be provided by libpthread which will come from host and ldso
> > > will complain when launching the native program on use
> > 
> > No, it won't. In a uninative build, a newer ld.so and libc.so are present so
> > the symbols will be found, we point at these using the dynamic-linker option
> > so the binary is linked to our own loader. This mechanism is used for all
> > the binaries we build, it just requires BUILD_LDFLAGS to be used.
> 
> I checked the logs of pcp-native build locally and its using it correctly
> 
> -Wl,--allow-shlib-undefined
> -Wl,--dynamic-linker=/mnt/b/yoe/master/build/tmp/sysroots-uninative/x86_64-
> linux/lib/ld-linux-x86-64.so.2
> 
> please note these options will instruct it to ignore undefined symbols
> during static linking phase. Secondly it will encode the INTERP correctly,
> which I hope will be edited to point to new installation path when it is
> reused from sstate on different host. Now, I am not sure if ld.so from
> uninative will load the libc.so and libpthread.so from uninative as well.
> Sadly, the failed build is wiped out from the worker (debian11-vk-3) so I am
> not able to peek into it. This has to re-occur sometime again

another thing to note is that these symbols moved from libpthread.so into libc.so
in glibc 2.32+,
Comment 6 Randy MacLeod 2025-02-27 15:37:45 UTC 
Randy will find someone to work on this.
Comment 7 Mingli Yu 2025-04-23 08:54:52 UTC 
I did the test on Ubuntu 20.04.6 with glibc 2.31.
$ cat /etc/issue
Ubuntu 20.04.6 LTS \n \l

$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.17) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

And use the sstate cache which generated on Ubuntu 22.04.4(whith glibc 2.35), but didn't reproduce the issue until now.
Comment 9 Randy MacLeod 2025-06-24 20:47:31 UTC 
Mingli is there a patch you can add so that if / when the problem happens again, we learn more about the bug?
Comment 10 Mingli Yu 2025-07-04 10:22:22 UTC 
I stil can't reproduce the issue locally and from the latest reproduce log https://errors.yoctoproject.org/Errors/Details/859030/, it seems it uses uninative linker as below:
-Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2

The full log as below:
2025-06-14 06:48:48,442 - oe-selftest - INFO -       | gcc  -isystem/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/include -O2 -pipe -I./src/include -I./src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds  -Wall -O2 -g   -I../src/include -I../src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds  -Wall -O2 -g   -I../../src/include -I../../src/include/pcp -DPCP_VERSION=\"6.3.7\" -fPIC -fno-strict-aliasing -D_GNU_SOURCE -Wshadow -Wno-array-bounds -DHAVE_TRACE -I../../src/pmdas/nvidia -Wall -O2 -g    -o newcontext newcontext.c -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L./src/libpcp/src -L./src/libpcp_web/src -L./src/libpcp_pmda/src -L./src/libpcp_mmv/src -L./src/libpcp_archive/src  -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L../src/libpcp/src -L../src/libpcp_web/src -L../src/libpcp_pmda/src -L../src/libpcp_mmv/src -L../src/libpcp_archive/src  -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -L/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -pthread -Wall -L../../src/libpcp/src -L../../src/libpcp_web/src -L../../src/libpcp_pmda/src -L../../src/libpcp_mmv/src -L../../src/libpcp_archive/src -L../../src/libpcp_gui/src -L../../src/libpcp_mmv/src -L../../src/libpcp_web/src -L../../src/libpcp_trace/src -L../../src/libpcp_import/src -lpcp
2025-06-14 06:48:48,442 - oe-selftest - INFO -       | collect2: error: ld returned 1 exit status


And then it prints the linker as "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld"
| /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld: /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/work/x86_64-linux/pcp-native/6.3.7/recipe-sysroot-native/usr/lib/liblzma.so.5: undefined reference to `pthread_join@GLIBC_2.34'
2025-06-14 08:14:38,494 - oe-selftest - INFO -       | collect2: error: ld returned 1 exit status

Not sure why the linker is "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleB-extended/tmp/hosttools/ld" though we pass "-Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2".

Maybe the file /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 is missing when build pcp-native as some job is cleaning the file /srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-oe/build-st/reproducibleA/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2?
Comment 11 Randy MacLeod 2026-01-13 22:42:51 UTC 
Mingli, I think this is another bug due to sharing of sstate-cache between distros. What two linux distros and detailed steps have you taken to try to reproduce the error ?
Comment 12 Randy MacLeod 2026-05-28 15:25:11 UTC 
Khem, Mingli, 

Is this still happening?
Do you have more details ?
Comment 13 Randy MacLeod 2026-06-11 15:00:36 UTC 
No info on whether this is still a problem, reopen if it is.
Comment 14 Khem Raj 2026-07-01 06:44:09 UTC 
(In reply to Randy MacLeod from comment #13)
> No info on whether this is still a problem, reopen if it is.

It still is a problem, got it on AB today.

https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/7464885/raw_inline
Comment 15 Paul Barker 2026-07-31 09:04:25 UTC 
Hi Khem,

Are we sure the recent failure is the same as the ones previously seen in 2025?

The error seen in 2025 was "collect2: error: ld returned 1 exit status", with undefined references at link time. This looks like a potential issue with sharing sstate across distros.

The recent build you link to shows two errors:
- pcp-native fails with "fatal error: readline/readline.h: No such file or directory"
- frr fails with clippy unable to load its shared libraries: "error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory"

These look like missing dependencies or race conditions during the build.

Perhaps we need to close this and open a new bug?
Comment 16 Khem Raj 2026-07-31 16:00:59 UTC 
(In reply to Paul Barker from comment #15)
> Hi Khem,
> 
> Are we sure the recent failure is the same as the ones previously seen in
> 2025?
> 
> The error seen in 2025 was "collect2: error: ld returned 1 exit status",
> with undefined references at link time. This looks like a potential issue
> with sharing sstate across distros.
> 
> The recent build you link to shows two errors:
> - pcp-native fails with "fatal error: readline/readline.h: No such file or
> directory"
> - frr fails with clippy unable to load its shared libraries: "error while
> loading shared libraries: libatomic.so.1: cannot open shared object file: No
> such file or directory"
> 
> These look like missing dependencies or race conditions during the build.
> 
> Perhaps we need to close this and open a new bug?

It's same problem now showing with different library. Symptoms is same where a native library added dependency based on host it was built on and sstate brought them over to the host that does not have the library