Bug 9131

This issue also happens on beaglebone and qemuppc but on qemumips64 it works without problem. I will update the results for qemuarm and qemumips once I test them.

This issue also happens on qemux86(32bit).

I started qemu serial and found the same warning:

qemux86 login: D-BUS per-session daemon address is: unix:abstract=/tmp/dbus-pSct7RtlI0,guid=ff64e80c2e1b6aea1fab483656c58c88
[57;205Rmatchbox: Cant find a keycode for keysym 269025056
matchbox: ignoring key shortcut XF86Calendar=!$contacts

matchbox: Cant find a keycode for keysym 2809
matchbox: ignoring key shortcut telephone=!$dates

matchbox: Cant find a keycode for keysym 269025050
matchbox: ignoring key shortcut XF86Start=!matchbox-remote -desktop

[settings daemon] Forking. run with -n to prevent fork

** (matchbox-panel:633): WARNING **: Failed to load applet "battery" (/usr/lib/matchbox-panel/libbattery.so: cannot open shared object file: No such file or directory).

** (matchbox-desktop:632): WARNING **: Cannot launch matchbox-terminal: Failed to read enough data from child pid pipe (No child processes)




For qemux86-64(64bit) I was able to start applications.

Additional information:
poky : 152914f2983c5d69001de1d46ce99547fa1e75fe.
Several BSP: MMax32 bit, genericx86.
Steps to reproduce it:
1. Compila an image or download it from AB.
2. Boot the image from SDCard or USB on MMax or desired platform.
3. Try to perform any application, such as terminal etc.
Expected Result: 
All the applications should be launched successfully.
Actual Result:
The applications are not being launched properly.

(In reply to comment #1)
> This issue also happens on beaglebone and qemuppc but on qemumips64 it works
> without problem. I will update the results for qemuarm and qemumips once I
> test them.

This issue also happens on qemuarm, but it works on qemumips

Interestingly, works on intel-core2-32.

core-image-sato also works for me on qemux86...

I'm stuck on the wrong end of a bad 3G link right now so can't download autobuilder images.  Hopefully on Monday my ADSL is back.

Doesn't appear to be specific to sysvinit, at least I can't replicate with sysv or systemd.

If I take an image from the autobuilder, it doesn't work.  Install libglib from the same release's rpm/ feed and it works.  I'm blaming prelink but I still can't replicate outside of the AB.

This issues is also reproducible on:
corei7-64 - Minnow max 32. Below you can find the AB link:
http://yocto-ab-master.jf.intel.com/pub/releases/yocto-2.1_M2.rc1/machines/intel-corei7-64/core-image-sato-sdk-intel-corei7-64.hddimg

(In reply to comment #9)
> This issues is also reproducible on:
> corei7-64 - Minnow max 32. Below you can find the AB link:
> http://yocto-ab-master.jf.intel.com/pub/releases/yocto-2.1_M2.rc1/machines/
> intel-corei7-64/core-image-sato-sdk-intel-corei7-64.hddimg

By mistake I added wrong information this is the related one:


This issue is also reproducible on core2-32 on Minnow max 32 bit, AB link:

http://yocto-ab-master.jf.intel.com/pub/releases/yocto-2.1_M2.rc1/machines/intel-core2-32/core-image-sato-sdk-intel-core2-32.hddimg.

meta-qt4 : 4058b0773381f894d915ea3dfac5fe052d82a9a7
meta-intel : aaab29ad036b13a0959925329b2b8e471869ca53
meta-qt3 : 7d958b928a4a38a186746fabbc0d8169dd8bb3a6
poky : 23056103c949b498c23b47579e8dd57ce78e6ed9

I still can't make an image built locally fail but if I take an autobuilder image, install prelink into it and run prelink -ua (remove all prelinking) and reboot it, Sato works correctly.

So, I'm blaming prelink somehow breaking the forking in glib's GSpawn API.

Mark, any thoughts?  The timing suggests that glibc 2.23 could be involved.

Richard can confirm that disabling image-prelink fixes the image.

Worked out why I can't replicate!

To replicate you need to have build without the security flags enabled, and with image-prelink enabled.  Having security flags enabled works, with or without prelink.

I've been bisecting this.  To replicate the problem you need:
1) Don't enable security flags.  They are disabled by default
2) Enable prelink
3) Have a commit past 8bedf767aa8fa9629dd126bdc649912cfe074f7f (glib-2.0: use the system libpcre)

I've no idea why glib-2.0 linking to the host libpcre instead of an internal fork would cause this problem, especially when that is the recommended linking method.

*Definitely* prelink.

Setting LD_USE_LOAD_BIAS=0 when starting matchbox-desktop fixes the problem, which basically disables prelink.

Mark, if you have any further ideas on how to chase this, I'm all ears!

I confirmed that "prelink -u /usr/bin/matchbox-desktop" also "fixes" the problem in a previously broken image.

Retitling to reflect the state of debugging.  We've disabled prelink in master for 2.1, rescheduling to 2.1M4 in case a simple fix is discovered, otherwise we can move this to 2.2.

Where we are at in debugging.

The sample application we have is setup to link to:

glib-2.0 and libc.so


glib-2.0 further requires libpcre and libpthread.


The problem is that both libc.so and libpthread contain a 'fork' function.  The former is a function, the later is an IFUNC.


When the prelinker parses the system, it parses the glib-2.0 library and internally defined the call to go to the IFUNC.  (Which is correct.)


When the prelinker parses the test application, it does NOT get a conflict warning from the libc.so (which binds symbols on a breadth-first basis.  This ends up binding the fork, called from glib to the libc.so version.  And then the application is now bound to the libc.so version.


When the program is executed, the dynamic loader runs and does some minimal relocation binding that is not possible in the prelinker -- it gets to the fork call and now the glib library binds to the IFUNC resolve function, but the application was told it's not an IFUNC -- so fork is never executed and a failure occurs.


This does NOT appear to be a bug in the prelinker or the glibc's ld.so.  As strange as it is, this type of dynamic linking can cause issues because the libraries are not fully qualified.  A failure should be able to happen without a prelinked library -- the prelinker just makes it significantly more likely to occur.


We believe that the problem is the application itself is not linked to both glib and libpthread (and libpcre) -- so no conflict is detected.  We're currently investigating updating the pkgconfig file for glib to include libpthread and libpcre, as they are both required.   (These are currently specified in the pkgconfig 'private' section, but it appears they should be promoted to the more public linking methods.)

Drinks all round at Wind River, thanks Mark.

Further information.  We believe we've tracked down the problem to the prelink-rtld.

If the application had a full set of DT_NEEDED, the issue would not have been a problem -- however an application is not required to have that.

Inside the prelink-rtld the system was doing effectively doing a "depth-first" search defining the scope and binding the symbols for resolution.

The ld.so rules indicate a "breadth-first" search should be preformed instead.  We have a standalone test case and a fix for the issue in progress.

Prelink commit: dc411f96f2e27bea69c1c1e5c1074451d796adfc appears to fix the issue.

Further testing in progress.

Excellent, thanks Mark and co!

I just want to update status on this.

We have a fix for the specific fork issue (and have for a few weeks now).  However, with the glibc version in master we've got other failures occurring in the test cases.

We're trying to identify and understand them before we go to reenable prelink.

Work is progressing, but this has stopped integration of this particular fix.

Cannot reproduce it, tested in below images 

intel-core2-32 - Minnow 32

http://yocto-ab-master.ostc.intel.com/pub/releases/yocto-meta-intel-5.0-krogoth-2.1.rc3/machines/intel-core2-32/core-image-sato-intel-core2-32.hddimg

intel-corei7-64 - NUC 

http://yocto-ab-master.ostc.intel.com/pub/releases/yocto-meta-intel-5.0-krogoth-2.1.rc3/machines/intel-corei7-64/core-image-sato-intel-corei7-64.hddimg


meta-qt4 : 4058b0773381f894d915ea3dfac5fe052d82a9a7
meta-intel : 91ffb8144c11c391ea3f62ce2a59a49e0f1cb01c
meta-qt3 : 7d958b928a4a38a186746fabbc0d8169dd8bb3a6
poky : 898a78357e72fb80f1f8e26ba90bad5b7b054a4f

You can't reproduce because we've disabled prelink until this is fixed.

As mentioned before.. we're still working on this issue.  Prelink undo is broken.  We think it's related to something in binutils, but until we know for sure we're not comfortable calling everything resolved.

A change in binutils 2.22 triggered a second fault in the prelinker.  Sections were not longer ordered by offset in some cases.  This caused a failure in the un-prelink validation stages.  We finally have figured out the cause and a possible solution to this issue.  It is in review.

There is one remaining test case failure that still needs to be looked at after this.

This bug has also been observed with use of mount.exfat from the fuse-exfat recipe in the meta-filesystem bitbake layer, armel target, prelink enabled.

Use of LD_USE_LOAD_BIAS=0 fixes the issue.

eg.
LD_USE_LOAD_BIAS=0 mount.exfat <my-device> <mount point>

The following link gives another explanation of this behaviour and a suggested fix.

https://sourceware.org/ml/libc-alpha/2016-05/msg00034.html

Austin - thank you for the link.  This confirms that the problems we are seeing are not unique to our configuration.

I will finish up the work to resolve the YP specific issue, and then focus some time on working with the libc folks to resolve it there as well.

Fixed in oe-core commits:


0b191e4bf0cba77669497d272dda46c536f7e527
9d2c82f7d3fc0fdafc5c4fdd1707324bc4cdbf22

*** Bug 8192 has been marked as a duplicate of this bug. ***