14404 2021-05-21 18:11:18 +0000 "Bitbake still alive" for an extremely long time (arm host pthread issue?) 2022-02-03 15:59:48 +0000 1 1 7 Build System, Metadata & Runtime OE-Core Scripts and Tools 3.3 x86 Multiple RESOLVED WORKSFORME AB-INT Medium normal 4.0 1 jon.mason ross.burton alexandre.belloni randy.macleod richard.purdie ross.burton tvgamblin --- --- Don't know oldest_to_newest 90463 0 jon.mason 2021-05-21 18:11:18 +0000 Multiple instances of bitbake running forever (until cancelled), with logs like: Bitbake still alive (5000s) Bitbake still alive (10000s) Bitbake still alive (15000s) Bitbake still alive (20000s) Bitbake still alive (25000s) Bitbake still alive (30000s) Examples are: https://autobuilder.yoctoproject.org/typhoon/#/builders/96/builds/1781/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1926/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1916/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2753/steps/12/logs/stdio 90495 1 randy.macleod 2021-05-27 14:48:06 +0000 First three AB reports are due to LTP hanging. Solved by disabling some LTP tests. The fourth is due to the python global interpreter lock only on Arm builder. Ross to check with some toolchain people he knows. 90496 2 ross.burton 2021-05-27 14:48:56 +0000 RP did some digging: Dumping state on this failure/lockup: https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2753 Was locked up in eSDK testing of core-image-minimal in execution of "devtool sdk-install meta-extsdk-toolchain". python was sitting in pthread_cond_wait() no other processes, no active fds that were interesting. In the end I broke the lock in gdb which resulted in: NOTE: Starting bitbake server... Loading cache...done. Loaded 0 entries from dependency cache. Parsing recipes...done. Parsing of 825 .bb files complete (0 cached, 825 parsed). 1459 targets, 53 skipped, 0 masked, 0 errors. INFO: meta-extsdk-toolchain is already installed Fatal Python error: drop_gil: PyCOND_WAIT(gil->switch_cond) failed Python runtime state: finalizing (tstate=0xaaab08055150) Current thread 0x0000ffffb6bae010 (most recent call first): File "/home/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/aarch64-pokysdk-linux/usr/lib/python3.9/_weakrefset.py", line 39 in _remove Aborted (core dumped) i.e. it was sitting waiting on the python GIL. Looks like a pthread bug. pthreads being used were: /home/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/aarch64-pokysdk-linux/lib/libpthread.so.0 i.e. *our* pthreads (and our python3.9). 92460 3 randy.macleod 2022-02-03 15:59:48 +0000 Only seen once so closing for now.