14761 2022-03-16 17:44:55 +0000 Unable to start bitbake server - previous bitbake process stuck in pthread_mutex_lock() 2022-04-16 22:07:37 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 3.1.15 x86 Multiple RESOLVED FIXED AB-INT Medium normal 3.1.16 1 steve steve alexandre.belloni poky.bs.watcher poky.watcher randy.macleod richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 92783 0 steve 2022-03-16 17:44:55 +0000 oe-selftest-centos centos7-ty-4 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3301 Discussion notes from irc: (07:07:48 AM) RP: sakoman: I've just had a look at the logs on the system (07:08:10 AM) RP: sakoman: what is interesting is that a test fails about 15:30 and yet the cooker says it exited cleanly (07:08:35 AM) RP: then we get all the resource unavailable (07:09:17 AM) RP: sakoman: 53363 ? S 0:00 /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/bin/python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/scripts/devtool finish devtool-upgrade-test1 meta-selftest (07:09:28 AM) RP: sakoman: there is a bitbake process hanging around :/ (07:09:53 AM) sakoman: RP: that is not good :-( (07:10:17 AM) RP: sakoman: the socket is still there and there is that process so it is blocked by that (07:11:04 AM) RP: sakoman: and it is stuck in pthread_mutex_lock() (07:12:00 AM) RP: it is stuck in a trace from a dlopen() (07:13:52 AM) RP: sadly it is using python 3.9 buildtools and we don't have the debug symbols in that 92784 1 steve 2022-03-16 17:46:11 +0000 Backtrace from worker provided by RP: (gdb) bt #0 0x00007fecb58dfdff in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libpthread.so.0 #1 0x00007fecb58d8fb0 in pthread_mutex_lock () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libpthread.so.0 #2 0x00007fecb5c9f3a3 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #3 0x00007fecb5c9c084 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #4 0x00007fecb5c9c9bb in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #5 0x00007fecb5ca71dc in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #6 0x00007fecb56e8051 in _dl_catch_exception () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 #7 0x00007fecb5ca6d4e in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #8 0x00007fecb58ca24c in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libdl.so.2 #9 0x00007fecb56e8051 in _dl_catch_exception () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 #10 0x00007fecb56e8103 in _dl_catch_error () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 #11 0x00007fecb58ca989 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libdl.so.2 #12 0x00007fecb58ca2ca in dlopen () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libdl.so.2 #13 0x00007fecb5ae3030 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #14 0x00007fecb5ab34fc in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #15 0x00007fecb5ab133d in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #16 0x00007fecb5a07ce4 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #17 0x00007fecb5990398 in _PyEval_EvalFrameDefault () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #18 0x00007fecb5a8e654 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #19 0x00007fecb59cdb7d in _PyFunction_Vectorcall () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #20 0x00007fecb5994b68 in _PyEval_EvalFrameDefault () 92795 2 randy.macleod 2022-03-17 14:37:49 +0000 Odd bug, wait until we see it again. is dlopen thread safe? Do we need to put a mutex around our calls to dlopen to protect against this lock. ctypes calls from utils.py may be where this is needed. 93079 3 richard.purdie 2022-04-14 16:32:05 +0000 I'd guess this is some variant of https://git.yoctoproject.org/poky/commit/?id=496cbc01ca0ffd2b582b9ab4c4e0f54db7b18b00 maybe from a warning message in the dl import. I'd suggest Steve backports disabling gc in that section and then we close this and see if it happens again. 93081 4 richard.purdie 2022-04-14 16:36:45 +0000 Also possibly related to https://git.yoctoproject.org/poky/commit/bitbake?id=928bcb10a46939eaf801bea0b633e4624b5b5dfa which is more painful to backport. We should try the gc fix at least though. 93082 5 steve 2022-04-14 17:01:08 +0000 Unfortunately https://git.yoctoproject.org/poky/commit/?id=496cbc01ca0ffd2b582b9ab4c4e0f54db7b18b00 changes a section of code that doesn't exist in 1.46 :-( That code is added in: https://git.yoctoproject.org/poky/commit/?id=928bcb10a46939eaf801bea0b633e4624b5b5dfa which also won't apply without some prior commits. Do you think I should keep pulling on this thread, or are we getting too invasive for something that has only happened once? 93084 6 richard.purdie 2022-04-14 21:08:45 +0000 (In reply to comment #5) > Unfortunately > https://git.yoctoproject.org/poky/commit/ > ?id=496cbc01ca0ffd2b582b9ab4c4e0f54db7b18b00 changes a section of code that > doesn't exist in 1.46 :-( > > That code is added in: > https://git.yoctoproject.org/poky/commit/ > ?id=928bcb10a46939eaf801bea0b633e4624b5b5dfa > > which also won't apply without some prior commits. > > Do you think I should keep pulling on this thread, or are we getting too > invasive for something that has only happened once? If you look closely at the diff you'll see the same code is there before and after that patch, specifically: with self.wlock: self.writer.send_bytes(obj) and we just need to put the disable/enable around the wlock. I think that much is worth doing in 1.46 and should be safe even if we have to tweak the patches slightly. 93111 7 richard.purdie 2022-04-16 22:07:37 +0000 https://git.yoctoproject.org/poky/commit/?h=dunfell&id=ce50594d709f473fbad30e187565531af6714aaa