Bug 12857

Summary: Running anything dotnet/CoreCLR based in a devshell doesn't work
Product: [Yocto Project Subprojects] Pseudo Reporter: Kai Ruhnau <kai.ruhnau>
Component: pseudoAssignee: Seebs <seebs>
Status: RESOLVED DUPLICATE QA Contact:
Severity: normal    
Priority: Medium+ CC: pauldotknopf, randy.macleod, yp.pseudo.watcher, yp.watcher
Version: unspecified   
Target Milestone: 2.8 M1   
Hardware: x86   
OS: x86_64   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Kai Ruhnau 2018-07-17 15:48:03 UTC
I have a Ubuntu 16.04 box with the dotnet tooling installed (https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/sdk-current) and can run the bare executable (dotnet) both natively and within the devshell.
However, as soon as I do something that actually boots the CoreCLR (like `dotnet --info`), the process runs into a deadlock when running with pseudo in a devshell.

I've seen this with both pyro and sumo.

Running with gdb, I can see six threads with the following (reduced) stack traces:

Thread 6 (Thread 0x7fffeeffd700 (LWP 17297)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff625b4e2 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
[… towards `clone`]

Thread 5 (Thread 0x7fffef7fe700 (LWP 17296)):
#0  0x00007ffff6be8197 in __GI___openat (fd=-100, file=0x6303c0 "/tmp/clr-debug-pipe-17271-726528498-in", oflag=0)
    at ../sysdeps/unix/sysv/linux/wordsize-64/../openat.c:58
#1  0x00007ffff7b8d753 in ?? ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#2  0x00007ffff7b8d9c9 in ?? ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#3  0x00007ffff7b92457 in open ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#4  0x00007ffff6169d7f in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
[… towards `clone`]

Thread 4 (Thread 0x7fffeffff700 (LWP 17295)):
#0  0x00007ffff6bec74d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff625d903 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
[… towards `clone`]

Thread 3 (Thread 0x7ffff4837700 (LWP 17294)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff7765dbd in __GI___pthread_mutex_lock (mutex=0x7ffff7dd40a0) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007ffff7b8ff7e in ?? ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#3  0x00007ffff7b9daf0 in fcntl ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#4  0x00007ffff5674dd4 in ustcomm_connect_unix_sock () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#5  0x00007ffff5679404 in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#6  0x00007ffff77636ba in start_thread (arg=0x7ffff4837700) at pthread_create.c:333
#7  0x00007ffff6bf841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7ffff5038700 (LWP 17293)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff567987c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#2  0x00007ffff77636ba in start_thread (arg=0x7ffff5038700) at pthread_create.c:333
#3  0x00007ffff6bf841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7ffff7fe0740 (LWP 17271)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff7765dbd in __GI___pthread_mutex_lock (mutex=0x7ffff7dd40a0) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007ffff7b8ff7e in ?? ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#3  0x00007ffff7b9ebef in fopen64 ()
   from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#4  0x00007ffff62435fa in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
[… repeat]
#15 0x00007ffff5e23acd in coreclr_initialize ()
   from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
#16 0x00007ffff65fb780 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libhostpolicy.so
[… repeat]
#19 0x00007ffff68a1cbf in ?? () from /usr/share/dotnet/host/fxr/2.1.1/libhostfxr.so
[… repeat]
#25 0x00007ffff68a1f0c in hostfxr_main_startupinfo () from /usr/share/dotnet/host/fxr/2.1.1/libhostfxr.so
#26 0x000000000040ac74 in ?? ()
#27 0x000000000040af05 in ?? ()
#28 0x00007ffff6b11830 in __libc_start_main (main=0x40ae60, argc=2, argv=0x7fffffffbd38, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffbd28) at ../csu/libc-start.c:291
#29 0x0000000000408a54 in _start ()
Comment 1 Randy MacLeod 2018-08-20 15:18:55 UTC
Peter, 
Can you determine if the problem is with pseudo, dotnet or if it's a irresistible force, immovable object conflict?

Kai, are you aware of any tricks that dotnet plays that would conflict with pseudo's design?
Comment 2 Seebs 2018-08-20 16:11:51 UTC
Hmm. So, looking at the threads, it looks like we've got an fopen64() call in some initialization code, and a clone call somewhere, and for some reason we've got stuff in lttng calling fcntl. Hmm.

Okay, hypothesis: Something's trying to do traces, which means that for it to allow some syscall to complete, it has to be able to run something else, and the something else also blocks. I suspect dotnet has tracing going on, maybe? Possibly it can be turned off somehow.
Comment 3 Kai Ruhnau 2018-08-21 09:36:36 UTC
On Linux, dotnet indeed uses lttng-ust for tracing and initializes it here very early:

https://github.com/dotnet/coreclr/blob/be7b9df6252593c918aaf7efa7f57f266349d179/src/pal/src/misc/tracepointprovider.cpp

The last line in the function suggests that making that dlopen fail has no ill effect and indeed creating an empty libcoreclrtraceptprovider.so still allows `dotnet --info` to run outside of pseudo. Just removing the file is detected at a later stage and renders the Shared Framework unusable, though. With an empty file, the lttng based stack traces are gone together with two threads. It's still dead-locking in the devshell.

I've followed the instructions from https://github.com/dotnet/symstore/blob/master/src/dotnet-symbol/README.md to get my hands on the debug symbols for better stack traces. The four remaining threads are then blocked at:
- https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/thread/threadsusp.cpp#L125
- https://github.com/dotnet/coreclr/blob/v2.1.1/src/debug/debug-pal/unix/twowaypipe.cpp#L90
- https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/synchmgr/synchmanager.cpp#L2209 (without pseudo in the stack trace)
- https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/thread/process.cpp#L2040
Comment 4 Paul Knopf 2018-08-31 18:52:04 UTC
Hey guys, I created another bug report that is similar to this issue, but I believe it has more relevant information.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12908

There is a unrelated bug in .NET Core that causes a hang when using lttng. I've removed it completely from my build and is reflected in the bug I reported.

Maybe we can migrate the discussion to the new bug to get rid of any static noise about lttng?

I also reported this issue with Microsoft.

https://github.com/dotnet/coreclr/issues/19682

Again, this issue doesn't have anything to do with lttng.
Comment 5 Randy MacLeod 2019-06-27 15:54:45 UTC
Seebs, do you agree that this is duplicate of 12908 ?
Comment 6 Seebs 2019-06-27 17:01:45 UTC
I think that looks very likely to be the same issue. Pseudo had a baked-in assumption that syscalls returned. In the specific case of opening a named pipe only for read or only for write, though, the call can block until the other end happens. As it happens, .NET relies on this for a special debug socket which is opened in a thread that then just waits for anything to happen. So this looks like the same thing -- fcntl is blocking because we've got a thread inside openat.c, calling underlying openat for "/tmp/clr-debug-pipe-...".
Comment 7 Randy MacLeod 2019-06-28 17:20:35 UTC
Believed to be a duplicate. Let us know if it isn't.

*** This bug has been marked as a duplicate of bug 12908 ***