Bug 3742

Summary: bitbake hangs forever when PR serv is enabled
Product: [Build System, Metadata & Runtime] BitBake Reporter: Martin Jansa <Martin.Jansa>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: VERIFIED FIXED QA Contact:
Severity: major    
Priority: Medium CC: alexandrux.palalau, jessica.zhang, poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: 1.4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---
Attachments:
Description Flags
strace+fd none

Description Martin Jansa 2013-01-17 11:32:06 UTC
Created attachment 1014 [details]
strace+fd

When building world which takes long time to finish I                                                                                                         
sometimes see bitbake hanging forever. It didn't happen                                                                                                       
until I've enabled PR Service. But maybe it's not caused                                                                                                      
by PR Service and it's just coincidence.                                                                                                                      
                                                                                                                                                              
I've to kill python processes manually and this traceback shown:                                                                                              
Exception in thread Thread-1:                                                                                                                                 
Traceback (most recent call last):                                                                                                                            
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner                                                                                      
    self.run()                                                                                                                                                
  File "/usr/lib/python2.7/threading.py", line 504, in run                                                                                                    
    self.__target(*self.__args, **self.__kwargs)                                                                                                              
  File "/home/jenkins/oe/shr-core-branches/shr-core/bitbake/lib/prserv/serv.py", line 173, in _work                                                           
    self.prserv.work_forever()                                                                                                                                
  File "/home/jenkins/oe/shr-core-branches/shr-core/bitbake/lib/prserv/serv.py", line 93, in work_forever                                                     
    self.handle_request()                                                                                                                                     
  File "/usr/lib/python2.7/SocketServer.py", line 265, in handle_request                                                                                      
    fd_sets = select.select([self], [], [], timeout)                                                                                                          
error: (4, 'Interrupted system call')

When this happens, there is only one python process doing something                                                                                           
for very long time:
31073 jenkins    20   0  4300   592   504 S  0.0  0.0  0:01.18 │  ├─ tee log.world.20130116_045210.log/bitbake.log
31072 jenkins    20   0  4392   612   588 S  0.0  0.0  0:00.00 │  └─ /bin/sh /home/jenkins/oe/shr-core-branches/shr-core/openembedded-core/scripts/bitbake -k 
31095 jenkins    20   0  210M 92628  4396 S  0.0  0.2  1:59.29 │     └─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
31104 jenkins    20   0  210M 92628  4396 S  0.0  0.2  0:08.43 │        ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
31103 jenkins    20   0 1031M  676M  3196 R  7.0  1.4  1h12:11 │        └─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
31106 jenkins    20   0 1031M  676M  3196 S  0.0  1.4  0:16.07 │           ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
31105 jenkins    20   0 1031M  676M  3196 S  0.0  1.4  2:00.04 │           ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
17586 jenkins    20   0 1031M  675M  1792 S  0.0  1.4  0:00.13 │           ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
 2400 jenkins    20   0 1031M  676M  3196 S  0.0  1.4  0:00.00 │           ├─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world
 2399 jenkins    20   0 1031M  676M  3196 S  0.0  1.4  0:00.00 │           └─ python /home/jenkins/oe/shr-core-branches/shr-core/bitbake/bin/bitbake -k world

17586, 2399, 2400, 31105
are waiting in futex
futex(0x7fb9f1475fc8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>

31106:
select(11, [10], [], [], {0, 500000})   = 0 (Timeout)
futex(0x6d8fc60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 10 -> socket:[414834257]
lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 11 -> /home/jenkins/oe/shr-core-branches/shr-core/cache/prserv.sqlite3

31095
select(4, [3], NULL, NULL, {0, 249996}) = 0 (Timeout)
lr-x------ 1 jenkins jenkins 64 Jan 17 03:23 3 -> pipe:[414838024]
l-wx------ 1 jenkins jenkins 64 Jan 17 03:23 4 -> pipe:[414838024]

31104
select(10, [9], [], [], {0, 500000})    = 0 (Timeout)
lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 9 -> socket:[414834253]
lrwx------ 1 jenkins jenkins 64 Jan 17 03:23 10 -> /home/jenkins/oe/shr-core-branches/shr-core/cache/prserv.sqlite3

31103:
lrwx------ 1 jenkins jenkins 64 Jan 17 03:14 6 -> socket:[414838023]
lrwx------ 1 jenkins jenkins 64 Jan 17 03:14 7 -> /home/jenkins/oe/shr-core-branches/shr-core/bitbake.lock
lr-x------ 1 jenkins jenkins 64 Jan 17 03:14 24 -> pipe:[429598906]
lr-x------ 1 jenkins jenkins 64 Jan 17 03:14 0 -> pipe:[414831254]

and looping this very fast:
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
select(7, [6], NULL, NULL, {0, 0})      = 0 (Timeout)
read(24, 0x1139be44, 102400)            = -1 EAGAIN (Resource temporarily unavailable)
wait4(-1, 0x7fff157c4110, WNOHANG, NULL) = 0

I'll attach 7s of strace from each python process + /proc/pid/fd from each. Is there some better way to debug which pipe is from where?

Usually last cooker output is rm_work_all in gimp, but I don't see
anything special in gimp recipe and it could be some other thread
in background.

NOTE: Running task 22874 of 23570 (ID: 996, /home/jenkins/oe/shr-core-branches/shr-core/meta-openembedded/meta-oe/recipes-graphics/gimp/gimp_2.8.2.bb,
do_rm_work_all)
NOTE: recipe gimp-2.8.2-r0: task do_rm_work_all: Started
NOTE: recipe gimp-2.8.2-r0: task do_rm_work_all: Succeeded
Exception in thread Thread-1:
.. traceback from above
Comment 1 Martin Jansa 2013-01-28 15:02:14 UTC
Interrupting bitbake with Ctrl+C also hangs now almost every time. Repeating Ctrl+C does not work. kill -9 from different screen window is needed :/.
Comment 2 Richard Purdie 2013-02-06 14:50:12 UTC
Interrupting bitbake when its doing which step of the build? When runuqueue is running spawning tasks? I just tried that here and it works fine for me :/
Comment 3 Richard Purdie 2013-02-07 00:23:57 UTC
This looks like its from bad interaction between the threading module and the rest of bitbake. Resolved in : http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=bdfc5207a0fdbed889c4a5997306adab82b1c2d4
Comment 4 Alexandru Palalau 2013-04-18 07:13:52 UTC
Verified on master branch.
Commit ID: dab237bc6a7679dfeaa981c14736aedc7b04ecb9