Created attachment 1014 [details] strace+fd When building world which takes long time to finish I sometimes see bitbake hanging forever. It didn't happen until I've enabled PR Service. But maybe it's not caused by PR Service and it's just coincidence. I've to kill python processes manually and this traceback shown: Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) File "/home/jenkins/oe/shr-core-branches/shr-core/bitbake/lib/prserv/serv.py", line 173, in _work self.prserv.work_forever() File "/home/jenkins/oe/shr-core-branches/shr-core/bitbake/lib/prserv/serv.py", line 93, in work_forever self.handle_request() File "/usr/lib/python2.7/SocketServer.py", line 265, in handle_request fd_sets = select.select([self], [], [], timeout) error: (4, 'Interrupted system call') When this happens, there is only one python process doing something for very long time: 31073 jenkins 20 0 4300 592 504 S 0.0 0.0 0:01.18 │ ├─ tee log.world.20130116_045210.log/bitbake.log 31072 jenkins 20 0 4392 612 588 S 0.0 0.0 0:00.00 │ └─ /bin/sh /home/jenkins/oe/shr-core-branches/shr-core/openembedded-core/scripts/bitbake -k 31095 jenkins 20 0 210M 92628 4396 S 0.0 0.2 1:59.29 │ └─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 31104 jenkins 20 0 210M 92628 4396 S 0.0 0.2 0:08.43 │ ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 31103 jenkins 20 0 1031M 676M 3196 R 7.0 1.4 1h12:11 │ └─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 31106 jenkins 20 0 1031M 676M 3196 S 0.0 1.4 0:16.07 │ ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 31105 jenkins 20 0 1031M 676M 3196 S 0.0 1.4 2:00.04 │ ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 17586 jenkins 20 0 1031M 675M 1792 S 0.0 1.4 0:00.13 │ ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 2400 jenkins 20 0 1031M 676M 3196 S 0.0 1.4 0:00.00 │ ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 2399 jenkins 20 0 1031M 676M 3196 S 0.0 1.4 0:00.00 │ └─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world 17586, 2399, 2400, 31105 are waiting in futex futex(0x7fb9f1475fc8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 31106: select(11, [10], [], [], {0, 500000}) = 0 (Timeout) futex(0x6d8fc60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 10 -> socket:[414834257] lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 11 -> /home/jenkins/oe/shr-core-branches/shr-core/cache/prserv.sqlite3 31095 select(4, [3], NULL, NULL, {0, 249996}) = 0 (Timeout) lr-x------ 1 jenkins jenkins 64 Jan 17 03:23 3 -> pipe:[414838024] l-wx------ 1 jenkins jenkins 64 Jan 17 03:23 4 -> pipe:[414838024] 31104 select(10, [9], [], [], {0, 500000}) = 0 (Timeout) lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 9 -> socket:[414834253] lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 10 -> /home/jenkins/oe/shr-core-branches/shr-core/cache/prserv.sqlite3 31103: lrwx------ 1 jenkins jenkins 64 Jan 17 03:14 6 -> socket:[414838023] lrwx------ 1 jenkins jenkins 64 Jan 17 03:14 7 -> /home/jenkins/oe/shr-core-branches/shr-core/bitbake.lock lr-x------ 1 jenkins jenkins 64 Jan 17 03:14 24 -> pipe:[429598906] lr-x------ 1 jenkins jenkins 64 Jan 17 03:14 0 -> pipe:[414831254] and looping this very fast: wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout) select(7, [6], NULL, NULL, {0, 0}) = 0 (Timeout) read(24, 0x1139be44, 102400) = -1 EAGAIN (Resource temporarily unavailable) wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0 I'll attach 7s of strace from each python process + /proc/pid/fd from each. Is there some better way to debug which pipe is from where? Usually last cooker output is rm_work_all in gimp, but I don't see anything special in gimp recipe and it could be some other thread in background. NOTE: Running task 22874 of 23570 (ID: 996, /home/jenkins/oe/shr-core-branches/shr-core/meta-openembedded/meta-oe/recipes-graphics/gimp/gimp_2.8.2.bb, do_rm_work_all) NOTE: recipe gimp-2.8.2-r0: task do_rm_work_all: Started NOTE: recipe gimp-2.8.2-r0: task do_rm_work_all: Succeeded Exception in thread Thread-1: .. traceback from above
Interrupting bitbake with Ctrl+C also hangs now almost every time. Repeating Ctrl+C does not work. kill -9 from different screen window is needed :/.
Interrupting bitbake when its doing which step of the build? When runuqueue is running spawning tasks? I just tried that here and it works fine for me :/
This looks like its from bad interaction between the threading module and the rest of bitbake. Resolved in : http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=bdfc5207a0fdbed889c4a5997306adab82b1c2d4
Verified on master branch. Commit ID: dab237bc6a7679dfeaa981c14736aedc7b04ecb9