I have a Ubuntu 16.04 box with the dotnet tooling installed (https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/sdk-current) and can run the bare executable (dotnet) both natively and within the devshell. However, as soon as I do something that actually boots the CoreCLR (like `dotnet --info`), the process runs into a deadlock when running with pseudo in a devshell. I've seen this with both pyro and sumo. Running with gdb, I can see six threads with the following (reduced) stack traces: Thread 6 (Thread 0x7fffeeffd700 (LWP 17297)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffff625b4e2 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… towards `clone`] Thread 5 (Thread 0x7fffef7fe700 (LWP 17296)): #0 0x00007ffff6be8197 in __GI___openat (fd=-100, file=0x6303c0 "/tmp/clr-debug-pipe-17271-726528498-in", oflag=0) at ../sysdeps/unix/sysv/linux/wordsize-64/../openat.c:58 #1 0x00007ffff7b8d753 in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #2 0x00007ffff7b8d9c9 in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #3 0x00007ffff7b92457 in open () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #4 0x00007ffff6169d7f in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… towards `clone`] Thread 4 (Thread 0x7fffeffff700 (LWP 17295)): #0 0x00007ffff6bec74d in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007ffff625d903 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… towards `clone`] Thread 3 (Thread 0x7ffff4837700 (LWP 17294)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007ffff7765dbd in __GI___pthread_mutex_lock (mutex=0x7ffff7dd40a0) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007ffff7b8ff7e in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #3 0x00007ffff7b9daf0 in fcntl () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #4 0x00007ffff5674dd4 in ustcomm_connect_unix_sock () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #5 0x00007ffff5679404 in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #6 0x00007ffff77636ba in start_thread (arg=0x7ffff4837700) at pthread_create.c:333 #7 0x00007ffff6bf841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7ffff5038700 (LWP 17293)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffff567987c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #2 0x00007ffff77636ba in start_thread (arg=0x7ffff5038700) at pthread_create.c:333 #3 0x00007ffff6bf841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7ffff7fe0740 (LWP 17271)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007ffff7765dbd in __GI___pthread_mutex_lock (mutex=0x7ffff7dd40a0) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007ffff7b8ff7e in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #3 0x00007ffff7b9ebef in fopen64 () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #4 0x00007ffff62435fa in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… repeat] #15 0x00007ffff5e23acd in coreclr_initialize () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so #16 0x00007ffff65fb780 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libhostpolicy.so [… repeat] #19 0x00007ffff68a1cbf in ?? () from /usr/share/dotnet/host/fxr/2.1.1/libhostfxr.so [… repeat] #25 0x00007ffff68a1f0c in hostfxr_main_startupinfo () from /usr/share/dotnet/host/fxr/2.1.1/libhostfxr.so #26 0x000000000040ac74 in ?? () #27 0x000000000040af05 in ?? () #28 0x00007ffff6b11830 in __libc_start_main (main=0x40ae60, argc=2, argv=0x7fffffffbd38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffbd28) at ../csu/libc-start.c:291 #29 0x0000000000408a54 in _start ()
Peter, Can you determine if the problem is with pseudo, dotnet or if it's a irresistible force, immovable object conflict? Kai, are you aware of any tricks that dotnet plays that would conflict with pseudo's design?
Hmm. So, looking at the threads, it looks like we've got an fopen64() call in some initialization code, and a clone call somewhere, and for some reason we've got stuff in lttng calling fcntl. Hmm. Okay, hypothesis: Something's trying to do traces, which means that for it to allow some syscall to complete, it has to be able to run something else, and the something else also blocks. I suspect dotnet has tracing going on, maybe? Possibly it can be turned off somehow.
On Linux, dotnet indeed uses lttng-ust for tracing and initializes it here very early: https://github.com/dotnet/coreclr/blob/be7b9df6252593c918aaf7efa7f57f266349d179/src/pal/src/misc/tracepointprovider.cpp The last line in the function suggests that making that dlopen fail has no ill effect and indeed creating an empty libcoreclrtraceptprovider.so still allows `dotnet --info` to run outside of pseudo. Just removing the file is detected at a later stage and renders the Shared Framework unusable, though. With an empty file, the lttng based stack traces are gone together with two threads. It's still dead-locking in the devshell. I've followed the instructions from https://github.com/dotnet/symstore/blob/master/src/dotnet-symbol/README.md to get my hands on the debug symbols for better stack traces. The four remaining threads are then blocked at: - https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/thread/threadsusp.cpp#L125 - https://github.com/dotnet/coreclr/blob/v2.1.1/src/debug/debug-pal/unix/twowaypipe.cpp#L90 - https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/synchmgr/synchmanager.cpp#L2209 (without pseudo in the stack trace) - https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/thread/process.cpp#L2040
Hey guys, I created another bug report that is similar to this issue, but I believe it has more relevant information. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12908 There is a unrelated bug in .NET Core that causes a hang when using lttng. I've removed it completely from my build and is reflected in the bug I reported. Maybe we can migrate the discussion to the new bug to get rid of any static noise about lttng? I also reported this issue with Microsoft. https://github.com/dotnet/coreclr/issues/19682 Again, this issue doesn't have anything to do with lttng.
Seebs, do you agree that this is duplicate of 12908 ?
I think that looks very likely to be the same issue. Pseudo had a baked-in assumption that syscalls returned. In the specific case of opening a named pipe only for read or only for write, though, the call can block until the other end happens. As it happens, .NET relies on this for a special debug socket which is opened in a thread that then just waits for anything to happen. So this looks like the same thing -- fcntl is blocking because we've got a thread inside openat.c, calling underlying openat for "/tmp/clr-debug-pipe-...".
Believed to be a duplicate. Let us know if it isn't. *** This bug has been marked as a duplicate of bug 12908 ***