9131 2016-02-17 21:09:29 +0000 Prelink breaks fork IFUNC 2016-12-15 16:25:26 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain 2.1 All Multiple RESOLVED FIXED High major 2.2 M3 8192 1 francisco.j.pedraza.gonzalez mark.hatle alexandru.c.georgescu alexandru.costinx.roman austin.phillips jose.perez.carranza mark.hatle meta.mr.watcher meta.watcher multisosnooley randy.macleod richard.purdie ross.burton sgw yi.zhao david.israelx.rodriguez.castellanos --- Regression (Used to work) No (bug/feature does not impact docs) oldest_to_newest 59168 0 francisco.j.pedraza.gonzalez 2016-02-17 21:09:29 +0000 After booting yocto image, SATO is not working properly. Any application can be opened successfully. If you go through terminal via serial mode, this message is displayed: ** (matchbox-desktop:807): WARNING **: Cannot launch "app name": Failed to read enough data from child pip pipe (No child processes). AB: http://autobuilder.yoctoproject.org/pub/releases/yocto-2.1_M2.rc3 59183 1 yi.zhao 2016-02-18 09:04:41 +0000 This issue also happens on beaglebone and qemuppc but on qemumips64 it works without problem. I will update the results for qemuarm and qemumips once I test them. 59186 2 alexandru.costinx.roman 2016-02-18 09:27:58 +0000 This issue also happens on qemux86(32bit). I started qemu serial and found the same warning: qemux86 login: D-BUS per-session daemon address is: unix:abstract=/tmp/dbus-pSct7RtlI0,guid=ff64e80c2e1b6aea1fab483656c58c88 [57;205Rmatchbox: Cant find a keycode for keysym 269025056 matchbox: ignoring key shortcut XF86Calendar=!$contacts matchbox: Cant find a keycode for keysym 2809 matchbox: ignoring key shortcut telephone=!$dates matchbox: Cant find a keycode for keysym 269025050 matchbox: ignoring key shortcut XF86Start=!matchbox-remote -desktop [settings daemon] Forking. run with -n to prevent fork ** (matchbox-panel:633): WARNING **: Failed to load applet "battery" (/usr/lib/matchbox-panel/libbattery.so: cannot open shared object file: No such file or directory). ** (matchbox-desktop:632): WARNING **: Cannot launch matchbox-terminal: Failed to read enough data from child pid pipe (No child processes) For qemux86-64(64bit) I was able to start applications. 59228 3 francisco.j.pedraza.gonzalez 2016-02-18 20:47:38 +0000 Additional information: poky : 152914f2983c5d69001de1d46ce99547fa1e75fe. Several BSP: MMax32 bit, genericx86. Steps to reproduce it: 1. Compila an image or download it from AB. 2. Boot the image from SDCard or USB on MMax or desired platform. 3. Try to perform any application, such as terminal etc. Expected Result: All the applications should be launched successfully. Actual Result: The applications are not being launched properly. 59244 4 yi.zhao 2016-02-19 07:20:10 +0000 (In reply to comment #1) > This issue also happens on beaglebone and qemuppc but on qemumips64 it works > without problem. I will update the results for qemuarm and qemumips once I > test them. This issue also happens on qemuarm, but it works on qemumips 59275 5 ross.burton 2016-02-19 15:54:32 +0000 Interestingly, works on intel-core2-32. 59302 6 ross.burton 2016-02-19 23:47:50 +0000 core-image-sato also works for me on qemux86... I'm stuck on the wrong end of a bad 3G link right now so can't download autobuilder images. Hopefully on Monday my ADSL is back. 59411 7 ross.burton 2016-02-23 21:15:19 +0000 Doesn't appear to be specific to sysvinit, at least I can't replicate with sysv or systemd. 59440 8 ross.burton 2016-02-24 17:41:00 +0000 If I take an image from the autobuilder, it doesn't work. Install libglib from the same release's rpm/ feed and it works. I'm blaming prelink but I still can't replicate outside of the AB. 59454 9 francisco.j.pedraza.gonzalez 2016-02-24 23:23:15 +0000 This issues is also reproducible on: corei7-64 - Minnow max 32. Below you can find the AB link: http://yocto-ab-master.jf.intel.com/pub/releases/yocto-2.1_M2.rc1/machines/intel-corei7-64/core-image-sato-sdk-intel-corei7-64.hddimg 59455 10 francisco.j.pedraza.gonzalez 2016-02-24 23:30:49 +0000 (In reply to comment #9) > This issues is also reproducible on: > corei7-64 - Minnow max 32. Below you can find the AB link: > http://yocto-ab-master.jf.intel.com/pub/releases/yocto-2.1_M2.rc1/machines/ > intel-corei7-64/core-image-sato-sdk-intel-corei7-64.hddimg By mistake I added wrong information this is the related one: This issue is also reproducible on core2-32 on Minnow max 32 bit, AB link: http://yocto-ab-master.jf.intel.com/pub/releases/yocto-2.1_M2.rc1/machines/intel-core2-32/core-image-sato-sdk-intel-core2-32.hddimg. meta-qt4 : 4058b0773381f894d915ea3dfac5fe052d82a9a7 meta-intel : aaab29ad036b13a0959925329b2b8e471869ca53 meta-qt3 : 7d958b928a4a38a186746fabbc0d8169dd8bb3a6 poky : 23056103c949b498c23b47579e8dd57ce78e6ed9 59588 11 ross.burton 2016-03-01 15:19:14 +0000 I still can't make an image built locally fail but if I take an autobuilder image, install prelink into it and run prelink -ua (remove all prelinking) and reboot it, Sato works correctly. So, I'm blaming prelink somehow breaking the forking in glib's GSpawn API. Mark, any thoughts? The timing suggests that glibc 2.23 could be involved. 59596 12 ross.burton 2016-03-01 17:05:30 +0000 Richard can confirm that disabling image-prelink fixes the image. 59606 13 ross.burton 2016-03-01 20:00:03 +0000 Worked out why I can't replicate! To replicate you need to have build without the security flags enabled, and with image-prelink enabled. Having security flags enabled works, with or without prelink. 59621 14 ross.burton 2016-03-02 13:01:32 +0000 I've been bisecting this. To replicate the problem you need: 1) Don't enable security flags. They are disabled by default 2) Enable prelink 3) Have a commit past 8bedf767aa8fa9629dd126bdc649912cfe074f7f (glib-2.0: use the system libpcre) I've no idea why glib-2.0 linking to the host libpcre instead of an internal fork would cause this problem, especially when that is the recommended linking method. 59640 15 ross.burton 2016-03-02 21:28:10 +0000 *Definitely* prelink. Setting LD_USE_LOAD_BIAS=0 when starting matchbox-desktop fixes the problem, which basically disables prelink. Mark, if you have any further ideas on how to chase this, I'm all ears! 59773 16 richard.purdie 2016-03-07 23:03:21 +0000 I confirmed that "prelink -u /usr/bin/matchbox-desktop" also "fixes" the problem in a previously broken image. 60011 17 ross.burton 2016-03-16 13:53:14 +0000 Retitling to reflect the state of debugging. We've disabled prelink in master for 2.1, rescheduling to 2.1M4 in case a simple fix is discovered, otherwise we can move this to 2.2. 60014 18 mark.hatle 2016-03-16 14:01:26 +0000 Where we are at in debugging. The sample application we have is setup to link to: glib-2.0 and libc.so glib-2.0 further requires libpcre and libpthread. The problem is that both libc.so and libpthread contain a 'fork' function. The former is a function, the later is an IFUNC. When the prelinker parses the system, it parses the glib-2.0 library and internally defined the call to go to the IFUNC. (Which is correct.) When the prelinker parses the test application, it does NOT get a conflict warning from the libc.so (which binds symbols on a breadth-first basis. This ends up binding the fork, called from glib to the libc.so version. And then the application is now bound to the libc.so version. When the program is executed, the dynamic loader runs and does some minimal relocation binding that is not possible in the prelinker -- it gets to the fork call and now the glib library binds to the IFUNC resolve function, but the application was told it's not an IFUNC -- so fork is never executed and a failure occurs. This does NOT appear to be a bug in the prelinker or the glibc's ld.so. As strange as it is, this type of dynamic linking can cause issues because the libraries are not fully qualified. A failure should be able to happen without a prelinked library -- the prelinker just makes it significantly more likely to occur. We believe that the problem is the application itself is not linked to both glib and libpthread (and libpcre) -- so no conflict is detected. We're currently investigating updating the pkgconfig file for glib to include libpthread and libpcre, as they are both required. (These are currently specified in the pkgconfig 'private' section, but it appears they should be promoted to the more public linking methods.) 60016 19 ross.burton 2016-03-16 14:07:29 +0000 Drinks all round at Wind River, thanks Mark. 60170 20 mark.hatle 2016-03-19 01:05:30 +0000 Further information. We believe we've tracked down the problem to the prelink-rtld. If the application had a full set of DT_NEEDED, the issue would not have been a problem -- however an application is not required to have that. Inside the prelink-rtld the system was doing effectively doing a "depth-first" search defining the scope and binding the symbols for resolution. The ld.so rules indicate a "breadth-first" search should be preformed instead. We have a standalone test case and a fix for the issue in progress. 60172 21 mark.hatle 2016-03-19 01:44:34 +0000 Prelink commit: dc411f96f2e27bea69c1c1e5c1074451d796adfc appears to fix the issue. Further testing in progress. 60174 22 ross.burton 2016-03-19 22:18:34 +0000 Excellent, thanks Mark and co! 61513 23 mark.hatle 2016-04-21 15:36:43 +0000 I just want to update status on this. We have a fix for the specific fork issue (and have for a few weeks now). However, with the glibc version in master we've got other failures occurring in the test cases. We're trying to identify and understand them before we go to reenable prelink. Work is progressing, but this has stopped integration of this particular fix. 62182 24 jose.perez.carranza 2016-05-13 15:25:28 +0000 Cannot reproduce it, tested in below images intel-core2-32 - Minnow 32 http://yocto-ab-master.ostc.intel.com/pub/releases/yocto-meta-intel-5.0-krogoth-2.1.rc3/machines/intel-core2-32/core-image-sato-intel-core2-32.hddimg intel-corei7-64 - NUC http://yocto-ab-master.ostc.intel.com/pub/releases/yocto-meta-intel-5.0-krogoth-2.1.rc3/machines/intel-corei7-64/core-image-sato-intel-corei7-64.hddimg meta-qt4 : 4058b0773381f894d915ea3dfac5fe052d82a9a7 meta-intel : 91ffb8144c11c391ea3f62ce2a59a49e0f1cb01c meta-qt3 : 7d958b928a4a38a186746fabbc0d8169dd8bb3a6 poky : 898a78357e72fb80f1f8e26ba90bad5b7b054a4f 62183 25 ross.burton 2016-05-13 15:28:33 +0000 You can't reproduce because we've disabled prelink until this is fixed. 62185 26 mark.hatle 2016-05-13 15:33:19 +0000 As mentioned before.. we're still working on this issue. Prelink undo is broken. We think it's related to something in binutils, but until we know for sure we're not comfortable calling everything resolved. 62836 27 mark.hatle 2016-06-09 15:41:30 +0000 A change in binutils 2.22 triggered a second fault in the prelinker. Sections were not longer ordered by offset in some cases. This caused a failure in the un-prelink validation stages. We finally have figured out the cause and a possible solution to this issue. It is in review. There is one remaining test case failure that still needs to be looked at after this. 65223 28 austin.phillips 2016-08-16 12:06:33 +0000 This bug has also been observed with use of mount.exfat from the fuse-exfat recipe in the meta-filesystem bitbake layer, armel target, prelink enabled. Use of LD_USE_LOAD_BIAS=0 fixes the issue. eg. LD_USE_LOAD_BIAS=0 mount.exfat <my-device> <mount point> The following link gives another explanation of this behaviour and a suggested fix. https://sourceware.org/ml/libc-alpha/2016-05/msg00034.html 65281 29 mark.hatle 2016-08-17 19:06:29 +0000 Austin - thank you for the link. This confirms that the problems we are seeing are not unique to our configuration. I will finish up the work to resolve the YP specific issue, and then focus some time on working with the libc folks to resolve it there as well. 65350 30 mark.hatle 2016-08-19 12:58:40 +0000 Fixed in oe-core commits: 0b191e4bf0cba77669497d272dda46c536f7e527 9d2c82f7d3fc0fdafc5c4fdd1707324bc4cdbf22 69162 31 edwin.plauchu.camacho 2016-12-15 16:25:26 +0000 *** Bug 8192 has been marked as a duplicate of this bug. ***