Created attachment 223 [details] source code of the test application Tree/Branch: poky/master Commit: dfce2695b96ab587b8235cfab2859a874e9d2eef Using the core-image-sato-sdk image of qemuarm, the attached lttng-ust demo application would result a segmentation fault in the target.
I can't reproduce the issue when I built the arm sato-sdk image on Ubuntu 11.04 x86-64 host using OE Build Configuration: BB_VERSION = "1.13.3" TARGET_ARCH = "arm" TARGET_OS = "linux-gnueabi" MACHINE = "qemuarm" DISTRO = "poky" DISTRO_VERSION = "1.0+snapshot-20110904" TUNE_FEATURES = "armv5 dsp thumb arm926ejs" TARGET_FPU = "soft" meta meta-yocto = "master:de5d52711bd810a63a0ad5260b799d6672952c32" However, autobuilder's arm sato-sdk image at http://autobuilder.yoctoproject.org/pub/nightly/20110903-1/machines/qemuarm/ does have this bug... This is somewhat weird...
(In reply to comment #1) > I can't reproduce the issue when I built the arm sato-sdk image on Ubuntu 11.04 > x86-64 host using > Now I can reproduce the issue using x86-64 Ubuntu 10.10 host.
(In reply to comment #2) > (In reply to comment #1) > > I can't reproduce the issue when I built the arm sato-sdk image on Ubuntu 11.04 > > x86-64 host using > > > Now I can reproduce the issue using x86-64 Ubuntu 10.10 host. Looks the issue only happens on qemuarm.
This seems to be related to prelink. If I remove prelink from USER_CLASSES in local.conf, the problem seems gone.
(In reply to comment #4) > This seems to be related to prelink. If I remove prelink from USER_CLASSES in > local.conf, the problem seems gone. Yes. I've the same phenomenon: removing image-prelink from USER_CLASSES makes the segfault go. (Previously I thought I couldn't reproduce it with Ubuntu11.04 -- actually it's because I disabled image-prelink by chance) So looks Mark could help to look into this bug. :-) BTW: looks the fix to bug 1473 may not help here since this bug happens to x86-64 host while bug 1473 happens on x86-32 host.
I had a report of a similar bug with QT on arm. The fix for 1473 also resolved the issue w/ QT on arm. Please try the latest version of prelink and see if it corrects the issue(s) you have had. If it does not, I can look at investigating it further.
(In reply to comment #6) > I had a report of a similar bug with QT on arm. The fix for 1473 also resolved > the issue w/ QT on arm. > > Please try the latest version of prelink and see if it corrects the issue(s) > you have had. If it does not, I can look at investigating it further. Hi Mark, I tried the fix for 1743 (http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=90ded12e485657f776fa9dac6567c9c84c28fa63), but still got the this bug. :-( A simpler way to reproduce the issue: On x86-64 Ubuntu host(e.g., 11.04 or 10.04), bitbake core-image-sato-sdk with MACHINE ?= "qemuarm" and USER_CLASSES ?= "image-prelink". later, run "runqemu qemuarm core-image-sato-sdk" to boot into target; in the target, gcc test.c -l ust && ./a.out, and you would see a segfault. The test.c is a simple C program: int main() { return 0; }. And, removing image-prelink from USER_CLASSES can make the segfault disappear.
(In reply to comment #7) > (In reply to comment #6) > > I had a report of a similar bug with QT on arm. The fix for 1473 also resolved > > the issue w/ QT on arm. > > > > Please try the latest version of prelink and see if it corrects the issue(s) > > you have had. If it does not, I can look at investigating it further. > > Hi Mark, > I tried the fix for 1743 Sorry for the typo: I meant 1473. > (http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=90ded12e485657f776fa9dac6567c9c84c28fa63), > but still got the this bug. :-(
From talking with some ust folks at Wind River, they indicated that in past versions there have been issues with ust, linker scripts and prelinking. I do not know if this has anything to do with the current failure, but it's something known to be an issue in the past.
We likely need to release with this defect. In the past problems like this have been in the linker or the linker scripts used by UST. I doubt we will have the ability to fully diagnose a resolve this issue prior to the release, so the workaround of avoiding the prelinker should be appropriate for the release.
A number of changes have occurred in the prelink which may resolve this issue. It should be retested.
(In reply to comment #11) > A number of changes have occurred in the prelink which may resolve this issue. > It should be retested. Hi Mark, I re-tested the bug based on today's poky master (commit 021848be9eeb71de8acfba6c66c11645b01523b8) and still got the same issue... :-( If I remove "image-prelink" from USER_CLASSES, there is no issue. So I think the bug is still there.
Estimate assumes significant debugging time is required.
Progress has finally been made on this issue. The following is an explanation of the problem from Donn Seeley: I spent about a day and a half on this one, and I figured out what's going on. The quick summary: It's an ARM libc bug that involves both TLS (thread-local storage) and prelinking. It should be extremely rare. The fix looks really simple. The Yocto bug log makes this defect sound like it's fairly general, but it's not. It's actually pretty specific to ARM and to the usermode-rcu package, which ust uses. Here is what appears to be happening: We build some random program and link it with -lust. When we run it, it dies with a SIGSEGV before reaching main(). Libust.so depends on liburcu-bp.so from the usermode-rcu package. Although libust.so is not prelinked, liburcu-bp.so IS prelinked; this is critical. Libust.so uses a TLS / __thread variable that is defined in liburcu- bp.so. There are special ARM-specific relocation types that allow two shared libraries to share thread-specific data. This is critical too. One more critical issue: although liburcu-bp.so is prelinked, we can't load it at its prelinked address, because we also link against librt.so, and librt.so uses that address. The dynamic linker is forced to relink liburcu-bp.so at a different address. In the course of relinking, it processes the special ARM relocation record mentioned above. The prelinker has already filled in the information, which is a short offset into a table of thread- specific data that is allocated per-thread for each library that uses TLS. Because the normal behavior of a relocation is to add the symbol value to an addend stored at the address being relocated, we end up adding the short offset to itself, doubling it. Now we have an awkward situation. The libust.so library doesn't know about the addend, so its TLS data for this element is correct. The liburcu-bp.so library has a different offset for the element. When we go to initialize the element for the first time in liburcu-bp.so, we write the address of the result at the doubled (broken) offset. Later, when we refer to the address from libust.so, we check the value at the correct offset, but it's NULL, so we eat hot SIGSEGV. The fix looks like it would pretty simple. Instead of doing: case R_ARM_TLS_DTPOFF32: if (sym != NULL) *reloc_addr += sym->st_value; break; ... we really want to do: case R_ARM_TLS_DTPOFF32: if (sym != NULL) *reloc_addr = sym->st_value; break; This allows us to ignore any addend when we re-link a prelinked library.
The fix suggestion worked for solving bug 2577 on qemuarm-sdk.
Fix applied in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e1913fbb01af52feee74bd6ab239678211c7ebb2