A build has been run on a new Fedora 43 worker on 27 November, and sadly it produced sstate that is causing issues on other distributions. Problematic sstate is being moved out into https://valkyrie.yocto.io/pub/shared-failure-data/fedora43-sstate/ Particularly: - rust-native/rust segfaults due to faulty llvm-native tracked separately in https://bugzilla.yoctoproject.org/show_bug.cgi?id=16058 - qemu-native/cargo-native/rustc-native/depmodwrapper-cross all can't be executed on alma8/rocky9 (at least) due to 'Inconsistency detected by ld.so: rtld.c: 1238: rtld_setup_main_map: Assertion `_dl_rtld_map.l_libname' failed!' I've started an a-full to see what remains https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2818
Correction: not depmodwrapper-cross, but kmod-native. I started another build: https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2820
As a note, a ticket also exists for the "Inconsistency detected" error: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16085
All known problematic sstate has been expunged from the cache, as indicated by https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2821 (and 2820 started earlier). The fails are for known, unrelated issues. As to exactly what Fedora 43 toolchain does to break things remains a mystery. Khem reproduced and started looking into it, but didn't get to the bottom yet.
*** Bug 16085 has been marked as a duplicate of this bug. ***
Khem, can you take ownership of this bug?
*** Bug 16124 has been marked as a duplicate of this bug. ***
We've seen similar problematic sstate built on Ubuntu 25.10 (see #16058). We can't be certain that both issues have the same cause, but it's something to investigate. Both Ubuntu 25.10 and Fedora 43 use binutils 2.45 and glibc 2.42, these are both pretty new. The issue is unlikely to be due to GCC 15 as that is also used in other distros. There is reason to suspect patchelf involvement - we've had issues before with this, and similar issues have been reported elsewhere (though not specifically for Ubuntu 25.10 / Fedora 43). If these distros build native binaries with a different layout (due to changes in the linker or default ELF linker script), that can change patchelf behaviour. For example, sections may be rearranged, holes may be introduced in binaries, etc, where previously they wouldn't be. That may then cause incorrect loading of these binaries on older distros. Some links: - https://github.com/NixOS/patchelf/pull/544, led me to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0da1d5002745cdc721bc018b582a8a9704d56c42. Older kernels without this patch may incorrectly load such binaries (https://bugzilla.kernel.org/show_bug.cgi?id=197921). - https://github.com/bminor/glibc/commit/20681be149b9eb1b6c1f4246bf4bd801221c86cd adds handling for segment gaps. This fix may not be present in earlier glibc versions. These are just hunches at the minute but I wanted to get things down before I forget them.
I spent a bit of time with this, some observations: 1. Sstate produced by f43 works on f43, and other new-ish distros, it's specifically centos 8/9 derivatives where the problem is. 2. To see the problem, make sure the centos 8/9 builds happen in a significantly different (and longer path) than the original f43 sstate. If it's exactly same path, patchelf will not write the new path to the interpreter into unpacked sstate. 3. The error is: Inconsistency detected by ld.so: rtld.c: 1238: rtld_setup_main_map: Assertion `_dl_rtld_map.l_libname' failed! The function name and line number matches up with tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 and glibc-2.42+git/elf/rtld.c Looking at the source code, it's going over sections and setting dl_rtld_map.l_libname only if PT_INTERP is in that iteration. So older glibc on centos 8/9 is perhaps not involved. 4. So the next step I'd like to take is to run this function under gdb, step by step, both in a working setting (f43), and in a broken setting (alma8), and figure out where the logic and data diverges. If you set up a build with exactly same (long) path on both distros, and take f43 sstate into it, the executable that is broken on centos8 and working on f43 will be exactly same sha256 (I checked). Now, weekend!
After RP installed gdb on the hosts (thanks!), I ran into the next hurdle: uninative has no debugging information so I can't set breakpoints or step through it. So I need to figure out how to rebuild it with debugging (and then install in place of the standard uninative).
TL;DR: alma8 kernel sets bogus phdr, almost certainly the needed fix is in comment 8 (thanks Paul). Longer version: I had finally managed to set up a proper debugging session on f43 and alma8, and here's what I see: F43 where things work and data looks sane: Breakpoint 1, dl_main (phdr=0x5555557f1000, phnum=15, user_entry=0x7fffffffda50, auxv=0x7fffffffdc78) at rtld.c:1358 (gdb) p *phdr $2 = {p_type = 6, p_flags = 4, p_offset = 2686976, p_vaddr = 2740224, p_paddr = 2740224, p_filesz = 840, p_memsz = 840, p_align = 8} Alma8 where phdr is bogus (also older gdb can't correctly calculate the sources, but nevermind): Breakpoint 2, dl_main (phdr=0x5555557e4000, phnum=15, user_entry=0x7fffffffd070, auxv=0x7fffffffd320) at /usr/src/debug/nativesdk-glibc/2.42+git/sysdeps/generic/dl-prop.h:1358 (gdb) p *phdr $28 = {p_type = 0, p_flags = 0, p_offset = 0, p_vaddr = 0, p_paddr = 0, p_filesz = 0, p_memsz = 0, p_align = 0} If I then print the correct phdr (as seen on f43), then the data is there: (gdb) p *(const Elf64_Phdr *) 0x5555557f1000 $27 = {p_type = 6, p_flags = 4, p_offset = 2686976, p_vaddr = 2740224, p_paddr = 2740224, p_filesz = 840, p_memsz = 840, p_align = 8}
I used git tag --contains to see where the fix first appeared, and it was in the 5.18 kernel. Centos 8 has 4.18, centos 9 has 5.14, so it definitely checks out.
I started looking at what patchelf is doing and lo and behold, the relevant code sequence in patchelf.cc was found almost immediately: /* Check if we can keep PHT at the beginning of the file. We'd like to do that, because it preverves compatibility with older kernels¹ - but if the PHT has grown too much, we have no other option but to move it at the end of the file. ¹ older kernels had a bug that prevented them from loading ELFs with PHDRs not located at the beginning of the file; it was fixed over 0da1d5002745cdc721bc018b582a8a9704d56c42 (2022-03-02) */ bool relocatePht = false; unsigned int i = 1; while (i < rdi(hdr()->e_shnum) && ((off_t) rdi(shdrs.at(i).sh_offset)) <= phtSize) { const auto & sectionName = getSectionName(shdrs.at(i)); if (!hasReplacedSection(sectionName) && !canReplaceSection(sectionName)) { relocatePht = true; break; } i++; } Not sure what to do next. Ideas?
Can we change the linker map in our native builds to have a larger PHT by default? or leave space after it, maybe pad it to a different boundary?
Alex is looking at this bug so make him the owner.
I got more debugging out of patchelf with PATCHELF_DEBUG=1 cmdline prefix. For an item build on alma8: ==== patching ELF file './qemu-x86_64' replacing section '.interp' with size 155 this is a dynamic library last page is 0x29d000 first page is 0x0 needed space is 2968 allocating new PT_LOAD segment rewriting sht from offset 0x28e9d0 to offset 0x290000 (size 1856) rewriting section '.note.gnu.property' from offset 0x388 (size 48) to offset 0x290740 (size 48) rewriting section '.note.gnu.build-id' from offset 0x3b8 (size 36) to offset 0x290770 (size 36) rewriting section '.interp' from offset 0x3dc (size 109) to offset 0x290798 (size 155) rewriting section '.gnu.hash' from offset 0x450 (size 864) to offset 0x290838 (size 864) rewriting symbol table section 1 writing ./qemu-x86_64 ==== For an item built on f43: ==== patching ELF file './qemu-x86_64' replacing section '.interp' with size 155 this is a dynamic library last page is 0x29d000 first page is 0x0 needed space is 3208 allocating new PT_LOAD segment rewriting pht from offset 0x40 to offset 0x290000 (size 1128) rewriting sht from offset 0x28ebb0 to offset 0x290468 (size 1920) rewriting section '.interp' from offset 0x174000 (size 109) to offset 0x290be8 (size 155) rewriting symbol table section 7 writing ./qemu-x86_64 ==== So the key difference seems to be the 'rewriting pht from offset 0x40 to offset 0x290000 (size 1128)' line. 'needed space' is bigger and that might be relevant too. Exactly why that decision is made isn't yet clear.
I spent more time debugging what triggers the decision to relocate pht. The two executable differ in how they order their sections (via objdump -x): alma8: Sections: Idx Name Size VMA LMA File off Algn 0 .note.gnu.property 00000030 0000000000000388 0000000000000388 00000388 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 1 .note.gnu.build-id 00000024 00000000000003b8 00000000000003b8 000003b8 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 2 .interp 0000006d 00000000000003dc 00000000000003dc 000003dc 2**0 CONTENTS, ALLOC, LOAD, READONLY, DATA 3 .gnu.hash 00000360 0000000000000450 0000000000000450 00000450 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 4 .dynsym 000033a8 00000000000007b0 00000000000007b0 000007b0 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 5 .dynstr 0000212f 0000000000003b58 0000000000003b58 00003b58 2**0 CONTENTS, ALLOC, LOAD, READONLY, DATA 6 .gnu.version 0000044e 0000000000005c88 0000000000005c88 00005c88 2**1 CONTENTS, ALLOC, LOAD, READONLY, DATA 7 .gnu.version_r 000001c0 00000000000060d8 00000000000060d8 000060d8 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA .... fedora43: Sections: Idx Name Size VMA LMA File off Algn 0 .note.gnu.build-id 00000024 0000000000000350 0000000000000350 00000350 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 1 .init 0000001b 0000000000000374 0000000000000374 00000374 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 2 .plt 00001d80 0000000000000390 0000000000000390 00000390 2**4 CONTENTS, ALLOC, LOAD, READONLY, CODE 3 .text 00170ec0 0000000000002140 0000000000002140 00002140 2**6 CONTENTS, ALLOC, LOAD, READONLY, CODE 4 .fini 0000000d 0000000000173000 0000000000173000 00173000 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 5 .interp 0000006d 0000000000174000 0000000000174000 00174000 2**0 CONTENTS, ALLOC, LOAD, READONLY, DATA 6 .gnu.hash 00000350 0000000000174070 0000000000174070 00174070 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 7 .dynsym 00003390 00000000001743c0 00000000001743c0 001743c0 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 8 .dynstr 0000212f 0000000000177750 0000000000177750 00177750 2**0 CONTENTS, ALLOC, LOAD, READONLY, DATA 9 .gnu.version 0000044c 0000000000179880 0000000000179880 00179880 2**1 CONTENTS, ALLOC, LOAD, READONLY, DATA Note that .init and other CODE sections come early. The loop iterates over the sections like this: while (i < rdi(hdr()->e_shnum) && ((off_t) rdi(shdrs.at(i).sh_offset)) <= phtSize) { const auto & sectionName = getSectionName(shdrs.at(i)); if (!hasReplacedSection(sectionName) && !canReplaceSection(sectionName)) { relocatePht = true; break; } i++; } For alma8 the loop goes over the first four sections (until the comparison with phtSize yields false), and none of them trigger the condition. For fedora43 the loop arrives at .init, the condition triggers, pht gets relocated. At this point I'm really reaching my maximum depth, but the key issue doesn't seem to be PHT size, but rather how f43 orders the sections.
So why is F43 putting sections in different order? This can be understood by running linker with --verbose: F43: [akanavin@fedora43-vk-1 ~]$ ld hello.o -lc --verbose GNU ld version 2.45-1.fc43 Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386pep i386pe elf64bpf using internal linker script: ================================================== /* Script for -z combreloc -z separate-code --rosegment */ ... Debian 13: alex@alex-lx-laptop:~$ ld hello.o -lc --verbose GNU ld (GNU Binutils for Debian) 2.44 Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386pep i386pe using internal linker script: ================================================== /* Script for -z combreloc -z separate-code */ ... Aha! --rosegment is missing. Why is that? Because Fedora configures binutils this way: https://src.fedoraproject.org/rpms/binutils/blob/main/f/binutils.spec#_777 CARGS="$CARGS --enable-rosegment=yes" At this point I would suggest we add --no-rosegment to native linker flags and close the issue, but... this enable-rosegment change has happened long before F43, and F42 also has it. And I did a private build with F42, and transferred it to alma8, and yes, the same issue occurs. The autobuilder has had an active F42 worker for a long time, and it has never contaminated sstate, so there's something else here that I don't yet fully understand.
Ka-boom! I just ran an experiment that confirms: Fedora 42 contaminates sstate in the same way as F43. 1. Add a tweak to qemu-native so that it - forces a rebuild - makes artifacts that won't contaminate other builds: --- a/meta/recipes-devtools/qemu/qemu-native_10.2.0.bb +++ b/meta/recipes-devtools/qemu/qemu-native_10.2.0.bb @@ -8,3 +8,6 @@ EXTRA_OECONF:append = " --target-list=${@get_qemu_usermode_target_list(d)} --dis PACKAGECONFIG ??= "pie" +do_install:append() { + echo "this is for alex fedora tests 1" > ${D}${bindir}/alex-test +} 2. Run a qemux86-64 build on fedora 42: https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3192 3. Run a qemuarm64 build on alma8 (it's important that it's a different build, to ensure build paths differ and patchelf has to modify the binaries): https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3111 Voila: a ton of failures, and it's the same 'Inconsistency detected by ld.so: rtld.c: 1238: rtld_setup_main_map: Assertion `_dl_rtld_map.l_libname' failed!' I still have no idea why this hasn't happened before, at some point f42 had to produce some native sstate that would be reused on alma/rocky.
So next thing is try a fix mentioned earlier: +++ b/meta/conf/bitbake.conf @@ -623,6 +623,7 @@ export BUILD_LDFLAGS = "-L${STAGING_LIBDIR_NATIVE} \ -Wl,-rpath-link,${STAGING_BASE_LIBDIR_NATIVE} \ -Wl,-rpath,${STAGING_LIBDIR_NATIVE} \ -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} \ + -Wl,--no-rosegment \ -Wl,-O1" And indeed, with this fix, and the same build sequence as in the previous comment, alma8 build runs to completion. I'll send a patch.
Someone pointed out that --no-rosegment cannot be enabled globally, as only new binutils (2.43 onwards?) support it. The docs for the option say: === --rosegment --no-rosegment Attempt to ensure that only a single read-only, non-code segment is created. Only useful when used in conjunction with the -z separate- code option. The resulting binaries should be smaller than if -z separate-code is used on its own. Without this option, or if --no-rosegment is specified, the -z separate-code option will create two read-only segments, one before the code segment and one after it. === So the next thing I tried was '-z noseparate-code'. In local tests it looked promising, as it also rearranges the sections in a way that doesn't trigger the issue, but in a build there are multiple failures: https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3208 So theoretically we could work around the failures, but looks like this option isn't very practical either.
Patch: https://lists.openembedded.org/g/openembedded-core/message/230620
Patch merged: https://git.openembedded.org/openembedded-core/commit/?id=bd89d1c0185c473ef77c8ae1dabf9f5ecb5abe76, closing this.