12857 2018-07-17 15:48:03 +0000 Running anything dotnet/CoreCLR based in a devshell doesn't work 2019-06-28 17:20:35 +0000 1 1 6 Yocto Project Subprojects Pseudo pseudo unspecified x86 x86_64 RESOLVED DUPLICATE 12908 Medium+ normal 2.8 M1 1 kai.ruhnau seebs pauldotknopf randy.macleod yp.pseudo.watcher yp.watcher --- --- Don't know oldest_to_newest 81127 0 kai.ruhnau 2018-07-17 15:48:03 +0000 I have a Ubuntu 16.04 box with the dotnet tooling installed (https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/sdk-current) and can run the bare executable (dotnet) both natively and within the devshell. However, as soon as I do something that actually boots the CoreCLR (like `dotnet --info`), the process runs into a deadlock when running with pseudo in a devshell. I've seen this with both pyro and sumo. Running with gdb, I can see six threads with the following (reduced) stack traces: Thread 6 (Thread 0x7fffeeffd700 (LWP 17297)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffff625b4e2 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… towards `clone`] Thread 5 (Thread 0x7fffef7fe700 (LWP 17296)): #0 0x00007ffff6be8197 in __GI___openat (fd=-100, file=0x6303c0 "/tmp/clr-debug-pipe-17271-726528498-in", oflag=0) at ../sysdeps/unix/sysv/linux/wordsize-64/../openat.c:58 #1 0x00007ffff7b8d753 in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #2 0x00007ffff7b8d9c9 in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #3 0x00007ffff7b92457 in open () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #4 0x00007ffff6169d7f in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… towards `clone`] Thread 4 (Thread 0x7fffeffff700 (LWP 17295)): #0 0x00007ffff6bec74d in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007ffff625d903 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… towards `clone`] Thread 3 (Thread 0x7ffff4837700 (LWP 17294)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007ffff7765dbd in __GI___pthread_mutex_lock (mutex=0x7ffff7dd40a0) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007ffff7b8ff7e in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #3 0x00007ffff7b9daf0 in fcntl () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #4 0x00007ffff5674dd4 in ustcomm_connect_unix_sock () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #5 0x00007ffff5679404 in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #6 0x00007ffff77636ba in start_thread (arg=0x7ffff4837700) at pthread_create.c:333 #7 0x00007ffff6bf841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7ffff5038700 (LWP 17293)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffff567987c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #2 0x00007ffff77636ba in start_thread (arg=0x7ffff5038700) at pthread_create.c:333 #3 0x00007ffff6bf841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7ffff7fe0740 (LWP 17271)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007ffff7765dbd in __GI___pthread_mutex_lock (mutex=0x7ffff7dd40a0) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007ffff7b8ff7e in ?? () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #3 0x00007ffff7b9ebef in fopen64 () from […]tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so #4 0x00007ffff62435fa in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so [… repeat] #15 0x00007ffff5e23acd in coreclr_initialize () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so #16 0x00007ffff65fb780 in ?? () from /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.1/libhostpolicy.so [… repeat] #19 0x00007ffff68a1cbf in ?? () from /usr/share/dotnet/host/fxr/2.1.1/libhostfxr.so [… repeat] #25 0x00007ffff68a1f0c in hostfxr_main_startupinfo () from /usr/share/dotnet/host/fxr/2.1.1/libhostfxr.so #26 0x000000000040ac74 in ?? () #27 0x000000000040af05 in ?? () #28 0x00007ffff6b11830 in __libc_start_main (main=0x40ae60, argc=2, argv=0x7fffffffbd38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffbd28) at ../csu/libc-start.c:291 #29 0x0000000000408a54 in _start () 81409 1 randy.macleod 2018-08-20 15:18:55 +0000 Peter, Can you determine if the problem is with pseudo, dotnet or if it's a irresistible force, immovable object conflict? Kai, are you aware of any tricks that dotnet plays that would conflict with pseudo's design? 81410 2 seebs 2018-08-20 16:11:51 +0000 Hmm. So, looking at the threads, it looks like we've got an fopen64() call in some initialization code, and a clone call somewhere, and for some reason we've got stuff in lttng calling fcntl. Hmm. Okay, hypothesis: Something's trying to do traces, which means that for it to allow some syscall to complete, it has to be able to run something else, and the something else also blocks. I suspect dotnet has tracing going on, maybe? Possibly it can be turned off somehow. 81411 3 kai.ruhnau 2018-08-21 09:36:36 +0000 On Linux, dotnet indeed uses lttng-ust for tracing and initializes it here very early: https://github.com/dotnet/coreclr/blob/be7b9df6252593c918aaf7efa7f57f266349d179/src/pal/src/misc/tracepointprovider.cpp The last line in the function suggests that making that dlopen fail has no ill effect and indeed creating an empty libcoreclrtraceptprovider.so still allows `dotnet --info` to run outside of pseudo. Just removing the file is detected at a later stage and renders the Shared Framework unusable, though. With an empty file, the lttng based stack traces are gone together with two threads. It's still dead-locking in the devshell. I've followed the instructions from https://github.com/dotnet/symstore/blob/master/src/dotnet-symbol/README.md to get my hands on the debug symbols for better stack traces. The four remaining threads are then blocked at: - https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/thread/threadsusp.cpp#L125 - https://github.com/dotnet/coreclr/blob/v2.1.1/src/debug/debug-pal/unix/twowaypipe.cpp#L90 - https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/synchmgr/synchmanager.cpp#L2209 (without pseudo in the stack trace) - https://github.com/dotnet/coreclr/blob/v2.1.1/src/pal/src/thread/process.cpp#L2040 81473 4 pauldotknopf 2018-08-31 18:52:04 +0000 Hey guys, I created another bug report that is similar to this issue, but I believe it has more relevant information. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12908 There is a unrelated bug in .NET Core that causes a hang when using lttng. I've removed it completely from my build and is reflected in the bug I reported. Maybe we can migrate the discussion to the new bug to get rid of any static noise about lttng? I also reported this issue with Microsoft. https://github.com/dotnet/coreclr/issues/19682 Again, this issue doesn't have anything to do with lttng. 84335 5 randy.macleod 2019-06-27 15:54:45 +0000 Seebs, do you agree that this is duplicate of 12908 ? 84337 6 seebs 2019-06-27 17:01:45 +0000 I think that looks very likely to be the same issue. Pseudo had a baked-in assumption that syscalls returned. In the specific case of opening a named pipe only for read or only for write, though, the call can block until the other end happens. As it happens, .NET relies on this for a special debug socket which is opened in a thread that then just waits for anything to happen. So this looks like the same thing -- fcntl is blocking because we've got a thread inside openat.c, calling underlying openat for "/tmp/clr-debug-pipe-...". 84342 7 randy.macleod 2019-06-28 17:20:35 +0000 Believed to be a duplicate. Let us know if it isn't. *** This bug has been marked as a duplicate of bug 12908 ***