| Summary: | Unable to start bitbake server - previous bitbake process stuck in pthread_mutex_lock() | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Steve Sakoman <steve> |
| Component: | bitbake | Assignee: | Steve Sakoman <steve> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | alexandre.belloni, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie |
| Version: | 3.1.15 | ||
| Target Milestone: | 3.1.16 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Steve Sakoman
2022-03-16 17:44:55 UTC
Backtrace from worker provided by RP: (gdb) bt #0 0x00007fecb58dfdff in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libpthread.so.0 #1 0x00007fecb58d8fb0 in pthread_mutex_lock () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libpthread.so.0 #2 0x00007fecb5c9f3a3 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #3 0x00007fecb5c9c084 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #4 0x00007fecb5c9c9bb in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #5 0x00007fecb5ca71dc in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #6 0x00007fecb56e8051 in _dl_catch_exception () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 #7 0x00007fecb5ca6d4e in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 #8 0x00007fecb58ca24c in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libdl.so.2 #9 0x00007fecb56e8051 in _dl_catch_exception () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 #10 0x00007fecb56e8103 in _dl_catch_error () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 #11 0x00007fecb58ca989 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libdl.so.2 #12 0x00007fecb58ca2ca in dlopen () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/lib/libdl.so.2 #13 0x00007fecb5ae3030 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #14 0x00007fecb5ab34fc in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #15 0x00007fecb5ab133d in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #16 0x00007fecb5a07ce4 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #17 0x00007fecb5990398 in _PyEval_EvalFrameDefault () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #18 0x00007fecb5a8e654 in ?? () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #19 0x00007fecb59cdb7d in _PyFunction_Vectorcall () from /home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/libpython3.9.so.1.0 #20 0x00007fecb5994b68 in _PyEval_EvalFrameDefault () Odd bug, wait until we see it again. is dlopen thread safe? Do we need to put a mutex around our calls to dlopen to protect against this lock. ctypes calls from utils.py may be where this is needed. I'd guess this is some variant of https://git.yoctoproject.org/poky/commit/?id=496cbc01ca0ffd2b582b9ab4c4e0f54db7b18b00 maybe from a warning message in the dl import. I'd suggest Steve backports disabling gc in that section and then we close this and see if it happens again. Also possibly related to https://git.yoctoproject.org/poky/commit/bitbake?id=928bcb10a46939eaf801bea0b633e4624b5b5dfa which is more painful to backport. We should try the gc fix at least though. Unfortunately https://git.yoctoproject.org/poky/commit/?id=496cbc01ca0ffd2b582b9ab4c4e0f54db7b18b00 changes a section of code that doesn't exist in 1.46 :-( That code is added in: https://git.yoctoproject.org/poky/commit/?id=928bcb10a46939eaf801bea0b633e4624b5b5dfa which also won't apply without some prior commits. Do you think I should keep pulling on this thread, or are we getting too invasive for something that has only happened once? (In reply to comment #5) > Unfortunately > https://git.yoctoproject.org/poky/commit/ > ?id=496cbc01ca0ffd2b582b9ab4c4e0f54db7b18b00 changes a section of code that > doesn't exist in 1.46 :-( > > That code is added in: > https://git.yoctoproject.org/poky/commit/ > ?id=928bcb10a46939eaf801bea0b633e4624b5b5dfa > > which also won't apply without some prior commits. > > Do you think I should keep pulling on this thread, or are we getting too > invasive for something that has only happened once? If you look closely at the diff you'll see the same code is there before and after that patch, specifically: with self.wlock: self.writer.send_bytes(obj) and we just need to put the disable/enable around the wlock. I think that much is worth doing in 1.46 and should be safe even if we have to tweak the patches slightly. |