Multiple instances of bitbake running forever (until cancelled), with logs like: Bitbake still alive (5000s) Bitbake still alive (10000s) Bitbake still alive (15000s) Bitbake still alive (20000s) Bitbake still alive (25000s) Bitbake still alive (30000s) Examples are: https://autobuilder.yoctoproject.org/typhoon/#/builders/96/builds/1781/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1926/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1916/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2753/steps/12/logs/stdio
First three AB reports are due to LTP hanging. Solved by disabling some LTP tests. The fourth is due to the python global interpreter lock only on Arm builder. Ross to check with some toolchain people he knows.
RP did some digging: Dumping state on this failure/lockup: https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2753 Was locked up in eSDK testing of core-image-minimal in execution of "devtool sdk-install meta-extsdk-toolchain". python was sitting in pthread_cond_wait() no other processes, no active fds that were interesting. In the end I broke the lock in gdb which resulted in: NOTE: Starting bitbake server... Loading cache...done. Loaded 0 entries from dependency cache. Parsing recipes...done. Parsing of 825 .bb files complete (0 cached, 825 parsed). 1459 targets, 53 skipped, 0 masked, 0 errors. INFO: meta-extsdk-toolchain is already installed Fatal Python error: drop_gil: PyCOND_WAIT(gil->switch_cond) failed Python runtime state: finalizing (tstate=0xaaab08055150) Current thread 0x0000ffffb6bae010 (most recent call first): File "/home/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/aarch64-pokysdk-linux/usr/lib/python3.9/_weakrefset.py", line 39 in _remove Aborted (core dumped) i.e. it was sitting waiting on the python GIL. Looks like a pthread bug. pthreads being used were: /home/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/aarch64-pokysdk-linux/lib/libpthread.so.0 i.e. *our* pthreads (and our python3.9).
Only seen once so closing for now.