Bug 14339

Summary:	bitbake generates zombie Parser processes (hard to reproduce)
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Yi Fan Yu <yf3yu>
Component:	bitbake	Assignee:	Randy MacLeod <randy.macleod>
Status:	RESOLVED WORKSFORME	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	poky.bs.watcher, poky.watcher, randy.macleod, yf3yu
Version:	unspecified
Target Milestone:	4.1 M2
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Yi Fan Yu 2021-04-07 14:20:36 UTC

Not able to reproduce consistently, i think this is also dependent on load on the shared machine, perhaps the signal handler doesn't get to run properly.

it usually happen when I hit control-c too many times and bitbake will hang.
I remember deleting the *sock and *lock files to be able to restart bitbake.


here is the top output maybe after 2 weeks (I have around 50 zombies laying around).
```
wrlbuild  6016  0.0  0.0      0     0 ?        Z    Apr05   0:00 [Parser-35] <defunct>   0     0 ?        ZN   Mar22   0:06 [Parser-31] <defunct>
yyu1      6266  0.0  0.0      0     0 ?        ZN   Mar22   0:08 [Parser-32] <defunct>
wrlbuild 25648  0.0  0.0      0     0 ?        Z    Apr05   0:00 [Parser-1] <defunct>
```
```
ps aux | grep 'Z' | wc -l
58
```

Comment 1 Yi Fan Yu 2021-04-08 14:50:17 UTC

10061 might be related

Need to figure out at which bitbake cycle are we getting stuck/hanging/needing to rm the lock file.

Comment 2 Yi Fan Yu 2021-04-13 20:17:11 UTC

Encountered a similar issue I believe:

```
yyu1@yow-lpggp3:/yow-lpggp32/yyu1/b/build$ MACHINE=intel-x86-64 bitbake lib32-clang-native                                                              
WARNING: You have included the meta-openstack layer, but 'openstack' has not been enabled in your DISTRO_FEATURES. Some bbappend fil.                   
Loading cache: 100% |#################################################################################################| Time: 0:00:01                   
Loaded 9617 entries from dependency cache.                                                                                                              
                                                                                                                                                        
Keyboard Interrupt, closing down...                                                                                                                     
                                                                                                                                                        
NOTE: No reply from server in 30s                                                                                                                       
Timeout while waiting for a reply from the bitbake server (60s) 
```


```
yyu1@yow-lpggp3:/yow-lpggp32/yyu1/b/build$ ls *ock
bitbake.lock  bitbake.sock
```

```
lsof bitbake.lock

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF        NODE NAME
Cooker   35965 yyu1    3u   REG   8,33        6 15328396296 bitbake.lock
Parser-2 36082 yyu1    3u   REG   8,33        6 15328396296 bitbake.lock
```

```
yyu1@yow-lpggp3:/yow-lpggp32/yyu1/b/build$ ps -aux | grep Z
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
yyu1     24750  0.0  0.0  13076  1052 pts/19   S+   16:00   0:00 grep Z
yyu1     36083  0.0  0.0      0     0 ?        Z    15:34   0:00 [Parser-3] <defunct>
```

Can't kill parser3, but the lock files do disappear if I kill parser-2 with
kill `kill -s SIGTERM 36082`

Comment 3 Yi Fan Yu 2021-04-13 20:38:49 UTC

Reproducible on this particular build server, but not reproducible on my personal machine.

Difference is that the system makes a slight pause after seeing this

```
Loading cache: 100% ###################################################################################################################################################| Time: 0:00:00
Loaded 1438 entries from dependency cache.
```

if you hit controlc right after "Loaded 1438 ...", the bitbake server will stop responding.

Comment 4 Yi Fan Yu 2021-04-19 20:28:28 UTC

only reproduced on poky today, not on oe-core or wrlinux.



yyu1@yow-lpggp3:/yow-lpggp32/yyu1/yocto/b_poky/build$ bitbake core-image-minimal
WARNING: You have included the meta-virtualization layer, but 'virtualization' has not .
Loading cache: 100% |####################################################| Time: 0:00:00
Loaded 4779 entries from dependency cache.                                     
                                                    
Keyboard Interrupt, closing down...                                                    
                                                                                 
NOTE: No reply from server in 30s                                                   
                                                           
Second Keyboard Interrupt, stopping...                                                              
                                                               
NOTE: No reply from server in 30s                                                              
Traceback (most recent call last):

Comment 5 Richard Purdie 2021-04-19 21:00:38 UTC

Could you put a time.sleep(5) in the code to isolate the exact spot we need the Ctrl+C to make it easier to reproduce?

Comment 6 Yi Fan Yu 2021-04-23 21:04:15 UTC

I spent some time adding time.sleep to locations.
And I have given up trying.
it sometimes stops responding and it happens at different steps...

usually it is able to recover.

Sorry for not being helpful

Comment 7 Randy MacLeod 2021-09-06 23:36:16 UTC

Added Yi Fan who may want to work on this bug now that he's got some time available.

Comment 8 Randy MacLeod 2022-04-28 15:23:12 UTC

Richard fixed a parse time race windown and shortened the lifespan of python zombies. Re-open if seen again.