14712 – build hangs: host make is not collecting its own children, turning them into zombies

Bug 14712 - build hangs: host make is not collecting its own children, turning them into zombies

Summary: build hangs: host make is not collecting its own children, turning them into ...

Status:	RESOLVED FIXED

Alias:	None

Product:	OE-Core
Classification:	Build System, Metadata & Runtime
Component:	configuration (show other bugs)
Version:	unspecified
Hardware:	x86 Multiple

Importance:	Medium+ normal
Target Milestone:	4.0 M4
Assignee:	Richard Purdie
QA Contact:

URL:
Whiteboard:	AB-INT

Depends on:
Blocks:

Reported:	2022-02-07 16:17 UTC by Alexander Kanavin
Modified:	2022-04-14 11:48 UTC (History)
CC List:	3 users (show)

See Also:
OS type for building Yocto:	---
Type of Regression:	---
Verified:
Documentation change:	No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Alexander Kanavin 2022-02-07 16:17:34 UTC

This typically happens when building a kernel:

    1 19877 19872 19872 ?           -1 Sl    6000  34:11 bitbake-server /home/pokybuild/yocto-worker/musl-qemux86/build/bitbake/bin/bitbake-server decafbad 3 5 /home/pokybuild/yocto-worker/musl-qemux86/build/build/bitbake-cookerdaemon.log /home/pokybuild/yocto-worker/musl-qemux86/build/build/bitbake.lock /home/pokybuild/yocto-worker/musl-qemux86/build/build/bitbake.sock 0 None 0
19877 24389 19872 19872 ?           -1 Sl    6000   3:30  \_ python3 /home/pokybuild/yocto-worker/musl-qemux86/build/bitbake/bin/bitbake-worker decafbad
24389 28152 28152 28152 ?           -1 SNs   6000   0:01  |   \_ python3 /home/pokybuild/yocto-worker/musl-qemux86/build/bitbake/bin/bitbake-worker decafbad
28152 29012 28152 28152 ?           -1 SN    6000   0:00  |       \_ /bin/sh /home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/temp/run.do_compile.28152
29012 29063 28152 28152 ?           -1 SN    6000   0:00  |           \_ make -j 16 -l 52 HOSTCC=gcc  HOSTCFLAGS=-isystem/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/usr/include -O2 -pipe HOSTLDFLAGS=-L/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/usr/lib                         -L/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 HOSTCPP=gcc  -E HOSTCXX=g++  HOSTCXXFLAGS=-isystem/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native/usr/include -O2 -pipe bzImage CC=i686-poky-linux-musl-gcc  -fuse-ld=bfd -fmacro-prefix-map=/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0=/usr/src/debug/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0=/usr/src/debug/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/qemux86-poky-linux-musl/linux-yocto/5.15.16+gitAUTOINC+26e884f8c2_f46816a75d-r0/recipe-sysroot-native=  -fdebug-prefix-map=/home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work-shared/qemux86/kernel-source=/usr/src/kernel LD=i686-poky-linux-musl-ld.bfd 
29063 56042 28152 28152 ?           -1 SN    6000   0:00  |               \_ make -f /home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work-shared/qemux86/kernel-source/scripts/Makefile.build obj=sound single-build= need-builtin=1 need-modorder=1
56042  3094 28152 28152 ?           -1 SN    6000   0:00  |                   \_ make -f /home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work-shared/qemux86/kernel-source/scripts/Makefile.build obj=sound/drivers need-builtin= need-modorder=1
 3094  3282 28152 28152 ?           -1 ZN    6000   0:00  |                       \_ [make] <defunct>


Most recent occurrence is
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/4696

Comment 1 Richard Purdie 2022-02-09 19:35:37 UTC

There are three other occurrences we've seen, all centos8, two kernel, one perl build:

https://lists.yoctoproject.org/g/swat/topic/another_hang_on_centos8_ty_1/86879896
https://lists.yoctoproject.org/g/swat/topic/hung_linux_yocto_build_in/86841587
https://lists.yoctoproject.org/g/swat/topic/info_for_swat_for_hung_build/86698083

Comment 2 Alexander Kanavin 2022-02-16 21:29:53 UTC

alma8:

      1  917008  917005  917005 ?             -1 Sl    6000   3:15 bitbake-server /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-server decafbad 3 5 /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st
 917008  944575  917005  917005 ?             -1 Sl    6000   0:29  \_ python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad
 944575  739396  739396  739396 ?             -1 SNs   6000   0:01  |   \_ python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad
 739396  740032  739396  739396 ?             -1 SN    6000   0:00  |       \_ /bin/sh /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-869978/tmp/work/qemux86_64-poky-linux/linux-yocto/5.15.22+gitAUTOINC+2d38a472b2_7f685244
 740032  740117  739396  739396 ?             -1 SN    6000   0:00  |           \_ make HOSTCC=gcc  HOSTCFLAGS=-isystem/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-869978/tmp/work/qemux86_64-poky-linux/linux-yocto/5.15.2
 740117 2623395  739396  739396 ?             -1 SN    6000   0:00  |               \_ make -f /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-869978/tmp/work-shared/qemux86-64/kernel-source/scripts/Makefile.build obj=fs si
2623395 2702144  739396  739396 ?             -1 SN    6000   0:00  |                   \_ make -f /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-869978/tmp/work-shared/qemux86-64/kernel-source/scripts/Makefile.build obj=f
2702144 2719059  739396  739396 ?             -1 ZN    6000   0:00  |                       \_ [sh] <defunct>

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3163

Comment 3 Richard Purdie 2022-03-31 21:48:25 UTC

Another instance of this on alma8 in linux-yocto inside an oe-selftest: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3379

opensuse153, centos8 and alma8 all have make 4.2.1. 

I'm starting to suspect this is something like:

https://git.savannah.gnu.org/cgit/make.git/commit/?id=78b5fec6898c26956d00548427cda1101cb80f8a
(from https://savannah.gnu.org/bugs/?51400)

Comment 4 Alexandre Belloni 2022-04-06 09:23:07 UTC

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/625/steps/12/logs/stdio

reproducible centos8-ty-1

Comment 5 Randy MacLeod 2022-04-11 17:53:46 UTC

If we can't convince upstream that this is a bug worth fixing, what is the plan for YP?

Comment 6 Richard Purdie 2022-04-13 15:09:45 UTC

https://bugs.centos.org/view.php?id=18432

Comment 7 Richard Purdie 2022-04-13 21:50:49 UTC

Patches to avoid the buggy make 4.2.1 on alma/centos/steam/opensuse are in review

Comment 8 Richard Purdie 2022-04-14 11:48:29 UTC

Disallowing make 4.2.1 on centos dervied and opensuse systems https://git.yoctoproject.org/poky/commit/?id=6bef16bde3a4e3d14f2b5a3c32b6697012544121