1443 2011-09-01 20:29:56 +0000 segfault of lttng-ust applications in qemuarm: caused by prelink: on x86-64 Ubuntu host 2012-10-25 15:00:17 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core unspecified x86 arm RESOLVED FIXED Medium normal 1.3 2577 1 lianhao.lu mark.hatle andrei.adrianx.dinu dexuan.cui jessica.zhang meta.mr.watcher meta.watcher raj.khem richard.purdie sgw --- --- --- oldest_to_newest 15959 0 223 lianhao.lu 2011-09-01 20:29:56 +0000 Created attachment 223 source code of the test application Tree/Branch: poky/master Commit: dfce2695b96ab587b8235cfab2859a874e9d2eef Using the core-image-sato-sdk image of qemuarm, the attached lttng-ust demo application would result a segmentation fault in the target. 16002 1 dexuan.cui 2011-09-03 23:01:39 +0000 I can't reproduce the issue when I built the arm sato-sdk image on Ubuntu 11.04 x86-64 host using OE Build Configuration: BB_VERSION = "1.13.3" TARGET_ARCH = "arm" TARGET_OS = "linux-gnueabi" MACHINE = "qemuarm" DISTRO = "poky" DISTRO_VERSION = "1.0+snapshot-20110904" TUNE_FEATURES = "armv5 dsp thumb arm926ejs" TARGET_FPU = "soft" meta meta-yocto = "master:de5d52711bd810a63a0ad5260b799d6672952c32" However, autobuilder's arm sato-sdk image at http://autobuilder.yoctoproject.org/pub/nightly/20110903-1/machines/qemuarm/ does have this bug... This is somewhat weird... 16013 2 dexuan.cui 2011-09-04 23:47:38 +0000 (In reply to comment #1) > I can't reproduce the issue when I built the arm sato-sdk image on Ubuntu 11.04 > x86-64 host using > Now I can reproduce the issue using x86-64 Ubuntu 10.10 host. 16107 3 dexuan.cui 2011-09-08 17:30:50 +0000 (In reply to comment #2) > (In reply to comment #1) > > I can't reproduce the issue when I built the arm sato-sdk image on Ubuntu 11.04 > > x86-64 host using > > > Now I can reproduce the issue using x86-64 Ubuntu 10.10 host. Looks the issue only happens on qemuarm. 16207 4 lianhao.lu 2011-09-14 23:23:37 +0000 This seems to be related to prelink. If I remove prelink from USER_CLASSES in local.conf, the problem seems gone. 16209 5 dexuan.cui 2011-09-15 00:51:49 +0000 (In reply to comment #4) > This seems to be related to prelink. If I remove prelink from USER_CLASSES in > local.conf, the problem seems gone. Yes. I've the same phenomenon: removing image-prelink from USER_CLASSES makes the segfault go. (Previously I thought I couldn't reproduce it with Ubuntu11.04 -- actually it's because I disabled image-prelink by chance) So looks Mark could help to look into this bug. :-) BTW: looks the fix to bug 1473 may not help here since this bug happens to x86-64 host while bug 1473 happens on x86-32 host. 16219 6 mark.hatle 2011-09-15 07:27:58 +0000 I had a report of a similar bug with QT on arm. The fix for 1473 also resolved the issue w/ QT on arm. Please try the latest version of prelink and see if it corrects the issue(s) you have had. If it does not, I can look at investigating it further. 16220 7 dexuan.cui 2011-09-15 08:14:25 +0000 (In reply to comment #6) > I had a report of a similar bug with QT on arm. The fix for 1473 also resolved > the issue w/ QT on arm. > > Please try the latest version of prelink and see if it corrects the issue(s) > you have had. If it does not, I can look at investigating it further. Hi Mark, I tried the fix for 1743 (http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=90ded12e485657f776fa9dac6567c9c84c28fa63), but still got the this bug. :-( A simpler way to reproduce the issue: On x86-64 Ubuntu host(e.g., 11.04 or 10.04), bitbake core-image-sato-sdk with MACHINE ?= "qemuarm" and USER_CLASSES ?= "image-prelink". later, run "runqemu qemuarm core-image-sato-sdk" to boot into target; in the target, gcc test.c -l ust && ./a.out, and you would see a segfault. The test.c is a simple C program: int main() { return 0; }. And, removing image-prelink from USER_CLASSES can make the segfault disappear. 16221 8 dexuan.cui 2011-09-15 08:16:17 +0000 (In reply to comment #7) > (In reply to comment #6) > > I had a report of a similar bug with QT on arm. The fix for 1473 also resolved > > the issue w/ QT on arm. > > > > Please try the latest version of prelink and see if it corrects the issue(s) > > you have had. If it does not, I can look at investigating it further. > > Hi Mark, > I tried the fix for 1743 Sorry for the typo: I meant 1473. > (http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=90ded12e485657f776fa9dac6567c9c84c28fa63), > but still got the this bug. :-( 16264 9 mark.hatle 2011-09-16 09:02:13 +0000 From talking with some ust folks at Wind River, they indicated that in past versions there have been issues with ust, linker scripts and prelinking. I do not know if this has anything to do with the current failure, but it's something known to be an issue in the past. 16316 10 mark.hatle 2011-09-20 08:18:54 +0000 We likely need to release with this defect. In the past problems like this have been in the linker or the linker scripts used by UST. I doubt we will have the ability to fully diagnose a resolve this issue prior to the release, so the workaround of avoiding the prelinker should be appropriate for the release. 17929 11 mark.hatle 2012-01-03 10:31:05 +0000 A number of changes have occurred in the prelink which may resolve this issue. It should be retested. 17967 12 dexuan.cui 2012-01-04 20:54:47 +0000 (In reply to comment #11) > A number of changes have occurred in the prelink which may resolve this issue. > It should be retested. Hi Mark, I re-tested the bug based on today's poky master (commit 021848be9eeb71de8acfba6c66c11645b01523b8) and still got the same issue... :-( If I remove "image-prelink" from USER_CLASSES, there is no issue. So I think the bug is still there. 21189 13 mark.hatle 2012-05-08 15:38:04 +0000 Estimate assumes significant debugging time is required. 23537 14 mark.hatle 2012-07-26 18:41:16 +0000 Progress has finally been made on this issue. The following is an explanation of the problem from Donn Seeley: I spent about a day and a half on this one, and I figured out what's going on. The quick summary: It's an ARM libc bug that involves both TLS (thread-local storage) and prelinking. It should be extremely rare. The fix looks really simple. The Yocto bug log makes this defect sound like it's fairly general, but it's not. It's actually pretty specific to ARM and to the usermode-rcu package, which ust uses. Here is what appears to be happening: We build some random program and link it with -lust. When we run it, it dies with a SIGSEGV before reaching main(). Libust.so depends on liburcu-bp.so from the usermode-rcu package. Although libust.so is not prelinked, liburcu-bp.so IS prelinked; this is critical. Libust.so uses a TLS / __thread variable that is defined in liburcu- bp.so. There are special ARM-specific relocation types that allow two shared libraries to share thread-specific data. This is critical too. One more critical issue: although liburcu-bp.so is prelinked, we can't load it at its prelinked address, because we also link against librt.so, and librt.so uses that address. The dynamic linker is forced to relink liburcu-bp.so at a different address. In the course of relinking, it processes the special ARM relocation record mentioned above. The prelinker has already filled in the information, which is a short offset into a table of thread- specific data that is allocated per-thread for each library that uses TLS. Because the normal behavior of a relocation is to add the symbol value to an addend stored at the address being relocated, we end up adding the short offset to itself, doubling it. Now we have an awkward situation. The libust.so library doesn't know about the addend, so its TLS data for this element is correct. The liburcu-bp.so library has a different offset for the element. When we go to initialize the element for the first time in liburcu-bp.so, we write the address of the result at the doubled (broken) offset. Later, when we refer to the address from libust.so, we check the value at the correct offset, but it's NULL, so we eat hot SIGSEGV. The fix looks like it would pretty simple. Instead of doing: case R_ARM_TLS_DTPOFF32: if (sym != NULL) *reloc_addr += sym->st_value; break; ... we really want to do: case R_ARM_TLS_DTPOFF32: if (sym != NULL) *reloc_addr = sym->st_value; break; This allows us to ignore any addend when we re-link a prelinked library. 25589 15 andrei.adrianx.dinu 2012-09-24 11:31:44 +0000 The fix suggestion worked for solving bug 2577 on qemuarm-sdk. 26911 16 richard.purdie 2012-10-25 15:00:17 +0000 Fix applied in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e1913fbb01af52feee74bd6ab239678211c7ebb2 223 2011-09-01 20:29:56 +0000 2011-09-01 20:29:56 +0000 source code of the test application lttng.c text/plain 227 lianhao.lu I2luY2x1ZGUgPHVzdC9tYXJrZXIuaD4KCmludCBtYWluKGludCBhcmdjLCBjaGFyICoqYXJndikK ewogIGludCB2PTY7CiAgY2hhciAqc3Q9Imx0dG5nIGV4YW1wbGUiOwoKICB0cmFjZV9tYXJrKG1h aW4sIG15ZXZlbnQsICJmaXJzdGFyZyAlZCBzZWNvbmRhcmcgJXMiLCB2LCBzdCk7CgogIHRyYWNl X21hcmsobWFpbiwgbXlvdGhlcmV2ZW50LCBNQVJLX05PQVJHUyk7CgogIHJldHVybiAwOwoKfQo=