Bug 14034

Created attachment 4719 [details]
build-cache.tar.gz

Contains:
build/cache
build/conf
bitbake-cookerdaemon.log

After manually "CTRL-C":
```
[ccxswbuild@ccxlc-ric-022 build]$time bitbake  -g brcm-zebu
WARNING: Duplicate inclusion for /tmp/rickliu/stingray-bcm958802a8046c/poky/build/conf/srcrevs.inc in /tmp/rickliu/stingray-bcm958802a8046c/poky/build/conf/local.conf
WARNING: Duplicate inclusion for /tmp/rickliu/stingray-bcm958802a8046c/poky/build/conf/srcrevs.inc in /tmp/rickliu/stingray-bcm958802a8046c/poky/build/conf/local.conf
WARNING: Duplicate inclusion for /tmp/rickliu/stingray-bcm958802a8046c/poky/build/conf/srcrevs.inc in /tmp/rickliu/stingray-bcm958802a8046c/poky/build/conf/local.conf
Loading cache: 100% |#########################################################################################################################################################################################################| Time: 0:00:02
Loaded 12105 entries from dependency cache.
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Exception during build_dependencies for do_deploy                                                     | ETA:  0:01:31
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Error during finalise of mc:crmu:/tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Exception during build_dependencies for do_deploy                                                     | ETA:  0:00:25
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Error during finalise of /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb: Exception during build_dependencies for do_compile                                                    | ETA:  0:00:18
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb: Error during finalise of mc:stingray2:/tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb: Exception during build_dependencies for do_compile
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb: Error during finalise of mc:crmu:/tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb: Exception during build_dependencies for do_compile
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb: Error during finalise of /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb
ERROR: ExpansionError during parsing /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb                                                                            | ETA:  0:00:17
Traceback (most recent call last):
  File "Var <CRMU_DEPLOYDIR>", line 1, in <module>
bb.data_smart.ExpansionError: Failure expanding variable CRMU_DEPLOYDIR, expression was ${@d.getVar('MCMACHINES').split()[1]} which triggered exception AttributeError: 'NoneType' object has no attribute 'split'


Keyboard Interrupt, closing down...


Second Keyboard Interrupt, stopping...

Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 604, in main
    event = eventHandler.waitEvent(0.25)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 609, in waitEvent
    self.eventQueueNotify.wait(delay)
  File "/usr/lib/python3.6/threading.py", line 551, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.6/threading.py", line 299, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 833, in main
    _, error = server.runCommand(["stateShutdown"])
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 346, in runCommand
    if not self.recv.poll(30):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 643, in poll
    return self.reader.poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 385, in bitbake_main
    configParams)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 837, in main
    state_force_shutdown()
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 821, in state_force_shutdown
    _, error = server.runCommand(["stateForceShutdown"])
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 346, in runCommand
    if not self.recv.poll(30):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 643, in poll
    return self.reader.poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt

real    179m50.159s
user    0m4.907s
sys     0m0.819s

```


and bitbake procsses are still alive:
```
[ccxswbuild@ccxlc-ric-022 build]$ps aux |grep bitbake |sort
ccxswbu+ 13301  0.0  0.0  14428  1036 pts/2    S+   11:51   0:00 /bin/grep bitbake
ccxswbu+  4893  0.1  0.6 830560 691904 ?       Sl   08:50   0:11 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4894  0.0  0.0 137752 69088 ?        S    08:50   0:00 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4904  0.0  0.5 718612 579500 ?       Sl   08:50   0:05 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4906  0.0  0.5 710100 571604 ?       Sl   08:50   0:06 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4907  0.0  0.5 717060 578188 ?       Sl   08:50   0:05 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4915  0.0  0.5 717208 578652 ?       Sl   08:50   0:04 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4920  0.0  0.5 713428 574944 ?       Sl   08:50   0:05 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4928  0.0  0.5 720448 581576 ?       Sl   08:50   0:04 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+  4933  0.0  0.5 718152 579068 ?       Sl   08:50   0:04 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu

```


and no new changes to bitbake-cookerdaemon.log or build/cache:
```
[ccxswbuild@ccxlc-ric-022 build]$ls -lort
total 4340
drwxr-xr-x 3 ccxswbuild    4096 Aug 18 15:54 buildhistory
drwxr-xr-x 3 ccxswbuild    4096 Sep  1 07:22 conf-stingray
drwxr-xr-x 3 ccxswbuild    4096 Sep  1 08:43 conf
drwxr-xr-x 3 ccxswbuild    4096 Sep  1 08:44 tmp-crmu
drwxr-xr-x 5 ccxswbuild    4096 Sep  1 08:44 tmp
srwxr-xr-x 1 ccxswbuild       0 Sep  1 08:50 bitbake.sock
-rw-r--r-- 1 ccxswbuild       5 Sep  1 08:50 bitbake.lock
srwxr-xr-x 1 ccxswbuild       0 Sep  1 08:50 hashserve.sock
-rw-r--r-- 1 ccxswbuild   67822 Sep  1 08:50 bitbake-cookerdaemon.log
drwxr-xr-x 2 ccxswbuild    4096 Sep  1 08:50 cache

```

Which revision of poky is this based upon? You say 3.1.3 so does that mean unreleased dunfell branch? or is this with master? Does the same issue occur with the latest bitbake master branch?

I'm using this tag 3.2_M2 from poky.git:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-poky?h=3.2_M2

poky.git 3.2_M2 tag 
meta-poky/conf/distro/poky.conf has version "3.1+snapshot-${DATE}":
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf?h=3.2_M2#n3

(In reply to comment #5)
> poky.git 3.2_M2 tag 
> meta-poky/conf/distro/poky.conf has version "3.1+snapshot-${DATE}":
> https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/
> poky.conf?h=3.2_M2#n3

Could you retest with recent master please? We're about to roll M3 and I think the bitbake server changes should fix this but am not 100% sure. I can't reproduce it with current master (which will become M3) at least.

ok i will try out the latest master branch

I've been able to reproduce this with master and with dunfell. It looks to be a problem in the cooker parsing when cancel_join_thread() is used which it is when there is a parsing failure.

I'm experimenting with different fixes.

A patch which seems to work is:

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 8baa58416a..5a289c6c17 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -2008,7 +2008,8 @@ class Parser(multiprocessing.Process):
             except queue.Empty:
                 pass
             else:
-                self.results.cancel_join_thread()
+                self.results.close()
+                self.results.join_thread()
                 break
 
             if pending:
@@ -2017,6 +2018,8 @@ class Parser(multiprocessing.Process):
                 try:
                     job = self.jobs.pop()
                 except IndexError:
+                    self.results.close()
+                    self.results.join_thread()
                     break
                 result = self.parse(*job)
                 # Clear the siggen cache after parsing to control memory usage, its huge
@@ -2136,10 +2139,11 @@ class CookerParser(object):
 
             bb.event.fire(event, self.cfgdata)
 
-        # Allow data left in the cancel queue to be discarded
-        self.parser_quit.cancel_join_thread()
         for process in self.processes:
             self.parser_quit.put(None)
+        self.parser_quit.close()
+        # Allow data left in the cancel queue to be discarded
+        self.parser_quit.cancel_join_thread()
 
         # Cleanup the queue before call process.join(), otherwise there might be
         # deadlocks.

Thank you for a quick patch.
I'll try to verify the patch soon.

But from my side,
the issue is happened randomly,
and must repeatedly run failed bitbake command to reproduce.
(I had once need to run bitbake 22 times to reproduce it)

Hi,

I just checkout to use master branch commit-a6709152e9788418e6c9431851437eb8ec4c6b46
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a6709152e9788418e6c9431851437eb8ec4c6b46

and applied your patch:
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index fc017dd19f..7d4cb48ce9 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1991,7 +1991,8 @@ class Parser(multiprocessing.Process):
             except queue.Empty:
                 pass
             else:
-                self.results.cancel_join_thread()
+                self.results.close()
+                self.results.join_thread()
                 break

             if pending:
@@ -2000,6 +2001,8 @@ class Parser(multiprocessing.Process):
                 try:
                     job = self.jobs.pop()
                 except IndexError:
+                    self.results.close()
+                    self.results.join_thread()
                     break
                 result = self.parse(*job)
                 # Clear the siggen cache after parsing to control memory usage, its huge
@@ -2119,10 +2122,11 @@ class CookerParser(object):

             bb.event.fire(event, self.cfgdata)

-        # Allow data left in the cancel queue to be discarded
-        self.parser_quit.cancel_join_thread()
         for process in self.processes:
             self.parser_quit.put(None)
+        self.parser_quit.close()
+        # Allow data left in the cancel queue to be discarded
+        self.parser_quit.cancel_join_thread()

         # Cleanup the queue before call process.join(), otherwise there might be
         # deadlocks.




But I'm still able to reporuduce the hang.

ERROR: ExpansionError during parsing /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/chimp/brcm-chimp.bb                                                                            | ETA:  0:04:41
Traceback (most recent call last):
  File "Var <CRMU_DEPLOYDIR>", line 1, in <module>
bb.data_smart.ExpansionError: Failure expanding variable CRMU_DEPLOYDIR, expression was ${@d.getVar('MCMACHINES').split()[1]} which triggered exception AttributeError: 'NoneType' object has no attribute 'split'


Keyboard Interrupt, closing down...


Second Keyboard Interrupt, stopping...

Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 617, in main
    event = eventHandler.waitEvent(0.25)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 648, in waitEvent
    self.eventQueueNotify.wait(delay)
  File "/usr/lib/python3.6/threading.py", line 551, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.6/threading.py", line 299, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 846, in main
    _, error = server.runCommand(["stateShutdown"])
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 364, in runCommand
    if not self.recv.poll(30):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 682, in poll
    return self.reader.poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 383, in bitbake_main
    configParams)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 850, in main
    state_force_shutdown()
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/ui/knotty.py", line 834, in state_force_shutdown
    _, error = server.runCommand(["stateForceShutdown"])
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 364, in runCommand
    if not self.recv.poll(30):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 682, in poll
    return self.reader.poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt





^CTraceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 364, in bitbake_main
    server_connection, ui_module = setup_bitbake(configParams)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 571, in connectProcessServer
    server_connection.connection.updateFeatureSet(featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 376, in updateFeatureSet
    _, error = self.runCommand(["setFeatures", featureset])
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 364, in runCommand
    if not self.recv.poll(30):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 682, in poll
    return self.reader.poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt
NOTE: Reconnecting to bitbake server...
^CTraceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 364, in bitbake_main
    server_connection, ui_module = setup_bitbake(configParams)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 571, in connectProcessServer
    server_connection.connection.updateFeatureSet(featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 376, in updateFeatureSet
    _, error = self.runCommand(["setFeatures", featureset])
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 364, in runCommand
    if not self.recv.poll(30):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 682, in poll
    return self.reader.poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt
NOTE: Reconnecting to bitbake server...
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#2)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#3)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#4)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
^CTraceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 364, in bitbake_main
    server_connection, ui_module = setup_bitbake(configParams)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 472, in setup_bitbake
    time.sleep(5)
KeyboardInterrupt
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#2)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#3)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#4)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 544, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
^CTraceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 364, in bitbake_main
    server_connection, ui_module = setup_bitbake(configParams)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 472, in setup_bitbake
    time.sleep(5)
KeyboardInterrupt
^C

but the bitbake hang after applied the patch seems different ...

I see these process alive from ps (which I didn't see before):
ccxswbu+ 20990  0.4  0.0 170904 35012 pts/0    Sl+  10:59   0:01 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+ 20992  2.7  0.6 788384 650888 ?       S    10:59   0:12 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 20996  0.0  0.0 134156 66216 ?        S    10:59   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 20999  0.8  0.5 677016 537924 ?       Sl   10:59   0:03 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21000  0.5  0.5 669848 531096 ?       Sl   10:59   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21001  0.2  0.0      0     0 ?        Z    10:59   0:01 [Parser-4] <defunct>
ccxswbu+ 21002  0.3  0.0      0     0 ?        Z    10:59   0:01 [Parser-5] <defunct>
ccxswbu+ 21003  0.4  0.5 669788 530948 ?       Sl   10:59   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21004  0.5  0.5 668508 529384 ?       Sl   10:59   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21005  0.2  0.0      0     0 ?        Z    10:59   0:01 [Parser-8] <defunct>
ccxswbu+ 21006  0.2  0.0      0     0 ?        Z    10:59   0:00 [Parser-9] <defunct>
ccxswbu+ 21008  0.7  0.5 671068 531960 ?       Sl   10:59   0:03 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21010  0.2  0.0      0     0 ?        Z    10:59   0:01 [Parser-11] <defunct>
ccxswbu+ 21014  0.5  0.0      0     0 ?        Z    10:59   0:02 [Parser-12] <defunct>
ccxswbu+ 21016  0.2  0.5 665948 526604 ?       Sl   10:59   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21018  0.1  0.0      0     0 ?        Z    10:59   0:00 [Parser-14] <defunct>
ccxswbu+ 21019  0.7  0.5 676188 537252 ?       Sl   10:59   0:03 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21022  0.0  0.5 664728 525212 ?       Sl   10:59   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21024  0.4  0.5 668764 529364 ?       Sl   10:59   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0

Created attachment 4720 [details]
log-after-patch.tar.gz

bitbake-cookerdaemon.log
cache/
cache/bb_codeparser.dat-21002
cache/bb_codeparser.dat-21014
cache/local_file_checksum_cache.dat-21006
cache/local_file_checksum_cache.dat-21001
cache/bb_codeparser.dat-21010
cache/hashserv.db-wal
cache/local_file_checksum_cache.dat-21005
cache/hashserv.db
cache/local_file_checksum_cache.dat-21002
cache/bb_persist_data.sqlite3
cache/bb_codeparser.dat-21005
cache/local_file_checksum_cache.dat-21010
cache/sanity_info
cache/bb_codeparser.dat-21018
cache/local_file_checksum_cache.dat-21014
cache/local_file_checksum_cache.dat
cache/bb_codeparser.dat-21001
cache/local_file_checksum_cache.dat-21018
cache/bb_codeparser.dat-21006
cache/hashserv.db-shm
cache/bb_codeparser.dat

Thanks, its unfortunate you're still seeing it :(. I'm not convinced we have the underlying problem fixed and haven't merged that patch to bitbake yet. I have recently merged a lot of debugging into master which improves the bitbske-cookerdaemon logging and fixes a few other minor bugs which may or may not contribute to the problem.

Can you reproduce with master with that extra debug and share the cookerdaemon logfile please?

Do you mean just checkout the latest poky.git master branch without applying your Comment-9 patch?

Sure, I could do it.

I found some issues with my patch and have updated it in master-next. If you could retry with poky's master-next branch and see if the issues is still present, that would be great, thanks!

I checkout master-next branch using commit-f5fe10489ebd74e468a123293524e6ab660fdd38.

It's still the same as last time (but different from the first hang report in this bug). 
The bitbake hangs.


running processes:
ccxswbu+ 16267  0.1  0.0 171084 35220 pts/0    Sl+  13:45   0:01 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+ 16269  1.0  0.6 778980 643476 ?       Sl   13:45   0:11 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16273  0.0  0.0 133660 65784 ?        S    13:45   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16283  0.1  0.5 664852 525764 ?       Sl   13:45   0:01 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16285  0.1  0.5 666132 527080 ?       Sl   13:45   0:01 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16287  0.2  0.5 672020 533644 ?       Sl   13:45   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16290  0.0  0.5 663316 524240 ?       Sl   13:45   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16292  0.2  0.5 669204 530668 ?       Sl   13:45   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16294  0.1  0.5 664084 525228 ?       Sl   13:45   0:01 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16296  0.1  0.5 666900 527812 ?       Sl   13:45   0:01 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16298  0.1  0.5 665364 526380 ?       Sl   13:45   0:01 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16299  0.2  0.5 670228 531208 ?       Sl   13:45   0:02 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/b/tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 16302  0.2  0.0      0     0 ?        Z    13:45   0:02 [Parser-16] <defunct>
ccxswbu+ 16303  0.0  0.0      0     0 ?        Z    13:45   0:01 [Parser-17] <defunct>




Console output:
Loading cache: 100% |###########################################################################################################################################################################| Time: 0:00:06
Loaded 11294 entries from dependency cache.
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Exception during build_dependencies for do_deploy                      | ETA:  --:--:--
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Error during finalise of mc:crmu:/tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb
ERROR: ExpansionError during parsing /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb                                              | ETA:  0:00:39
Traceback (most recent call last):
  File "Var <CRMU_DEPLOYDIR>", line 1, in <module>
bb.data_smart.ExpansionError: Failure expanding variable CRMU_DEPLOYDIR, expression was ${@d.getVar('MCMACHINES').split()[1]} which triggered exception AttributeError: 'NoneType' object has no attribute 'split'


Keyboard Interrupt, closing down...

NOTE: No reply from server in 30s
Timeout while waiting for a reply from the bitbake server (60s)

----------------------
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#3)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#4)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#5)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#6)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#7)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#8)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).

I think Joshua and I tracked down another cause of problems. The fix was merged into master:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=42bbf1451ac67675de52f7dc0285b1f3bb24216b

Would you be able to retest with that included? Sorry about the testing requests but it is important we ensure we have fixed all the issues here if we can.

Yes, I can do the test using commit-42bbf1451ac67675de52f7dc0285b1f3bb24216b:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14034#add_comment

I'll get back to you later with test results.

At commit-42bbf1451ac67675de52f7dc0285b1f3bb24216b,
I got bitbake hang after the 4th retry.

```
ccxswbu+ 18431  0.4  0.0 172944 37076 pts/0    Sl+  16:07   0:01 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu
ccxswbu+ 18492  2.8  0.5 658976 523348 ?       Sl   16:07   0:10 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 18854  0.0  0.0 131568 63536 ?        S    16:07   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21728  0.2  0.4 575100 435832 ?       Sl   16:08   0:01 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 21731  0.3  0.0      0     0 ?        Z    16:08   0:01 [Parser-6] <defunct>
ccxswbu+ 21734  0.3  0.0      0     0 ?        Z    16:08   0:01 [Parser-7] <defunct>
ccxswbu+ 21738  0.4  0.0      0     0 ?        Z    16:08   0:01 [Parser-8] <defunct>
ccxswbu+ 21744  0.5  0.0      0     0 ?        Z    16:08   0:01 [Parser-9] <defunct>
ccxswbu+ 21755  0.5  0.0      0     0 ?        Z    16:08   0:01 [Parser-10] <defunct>
ccxswbu+ 21762  0.3  0.0      0     0 ?        Z    16:08   0:01 [Parser-11] <defunct>
ccxswbu+ 21782  0.5  0.0      0     0 ?        Z    16:08   0:02 [Parser-12] <defunct>
ccxswbu+ 21790  0.2  0.0      0     0 ?        Z    16:08   0:01 [Parser-13] <defunct>
ccxswbu+ 21797  0.3  0.0      0     0 ?        Z    16:08   0:01 [Parser-14] <defunct>
ccxswbu+ 21804  0.4  0.0      0     0 ?        Z    16:08   0:01 [Parser-15] <defunct>
ccxswbu+ 21811  0.5  0.0      0     0 ?        Z    16:08   0:02 [Parser-16] <defunct>
ccxswbu+ 21818  0.5  0.0      0     0 ?        Z    16:08   0:02 [Parser-17] <defunct>
```

console-output:
```
Loaded 9054 entries from dependency cache.
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Exception during build_dependencies for do_deploy            | ETA:  --:--:--
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Error during finalise of mc:crmu:/tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Exception during build_dependencies for do_deploy             | ETA:  0:03:20
WARNING: /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb: Error during finalise of /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb
ERROR: ExpansionError during parsing /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb                                    | ETA:  0:01:44
Traceback (most recent call last):
  File "Var <CRMU_DEPLOYDIR>", line 1, in <module>
bb.data_smart.ExpansionError: Failure expanding variable CRMU_DEPLOYDIR, expression was ${@d.getVar('MCMACHINES').split()[1]} which triggered exception AttributeError: 'NoneType' object has no attribute 'split'


Keyboard Interrupt, closing down...

NOTE: No reply from server in 30s
Timeout while waiting for a reply from the bitbake server (60s)
return=1
```

Created attachment 4724 [details]
0917-bitbake-cookerdaemon.log

0917-bitbake-cookerdaemon.log

Another trial:
In a fresh clean poky/build folder,
right after ctrl-c to interrupt the bitbake hang,
start another "bitbake -g".

ps aux output:
```
ccxswbu+ 38388  2.4  0.6 749840 614104 ?       Sl   16:30   0:10 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 38392  0.0  0.0 130084 62348 ?        S    16:30   0:00 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 38404  1.2  0.5 648536 510036 ?       Sl   16:30   0:05 bitbake-server /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake-server decafbad 3 5 /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake-cookerdaemon.log /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.lock /tmp/rickliu/stingray-bcm958802a8046c/poky/build/bitbake.sock None None 0
ccxswbu+ 38405  1.1  0.0      0     0 ?        Z    16:30   0:05 [Parser-10] <defunct>
ccxswbu+ 38406  1.0  0.0      0     0 ?        Z    16:30   0:04 [Parser-11] <defunct>
ccxswbu+ 38409  0.7  0.0      0     0 ?        Z    16:30   0:03 [Parser-12] <defunct>
ccxswbu+ 38412  0.9  0.0      0     0 ?        Z    16:30   0:04 [Parser-13] <defunct>
ccxswbu+ 38414  0.8  0.0      0     0 ?        Z    16:30   0:03 [Parser-14] <defunct>
ccxswbu+ 38416  0.7  0.0      0     0 ?        Z    16:30   0:03 [Parser-15] <defunct>
ccxswbu+ 38419  0.7  0.0      0     0 ?        Z    16:30   0:03 [Parser-16] <defunct>
ccxswbu+ 38421  0.9  0.0      0     0 ?        Z    16:30   0:04 [Parser-17] <defunct>

# start another "bitbake -g":
ccxswbu+ 38827  0.2  0.0 234732 28876 pts/0    Sl+  16:36   0:00 python3 /tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/bin/bitbake -g brcm-zebu

```


console output:
PS:
return=1 is my echo "return=$? after "bitbake -g" command
```
ERROR: ExpansionError during parsing /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb                                    | ETA:  0:00:30
Traceback (most recent call last):
  File "Var <CRMU_DEPLOYDIR>", line 1, in <module>
bb.data_smart.ExpansionError: Failure expanding variable CRMU_DEPLOYDIR, expression was ${@d.getVar('MCMACHINES').split()[1]} which triggered exception AttributeError: 'NoneType' object has no attribute 'split'


Keyboard Interrupt, closing down...

NOTE: No reply from server in 30s
Timeout while waiting for a reply from the bitbake server (60s)
return=1

===============
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#1)...
NOTE: Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE: Retrying server connection (#2)...
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#3)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#4)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#5)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#6)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#7)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#8)... (Traceback (most recent call last):
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/main.py", line 453, in setup_bitbake
    server_connection = bb.server.process.connectProcessServer(sockname, featureset)
  File "/tmp/rickliu/stingray-bcm958802a8046c/poky/bitbake/lib/bb/server/process.py", line 572, in connectProcessServer
    sock.connect(os.path.basename(sockname))
BlockingIOError: [Errno 11] Resource temporarily unavailable
)
ERROR: Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).
return=1
```

Created attachment 4725 [details]
0917-bitbake-cookerdaemon-02.log

0917-bitbake-cookerdaemon-02.log

Starting another bitbake -g will give those connection errors if the original has hung so that isn't surprising.

I think the originally reported bug here is fixed (bitbake -g hanging) but the report has changed to Ctrl+C interrupting a failing parse can hang?

Neither Joshua or I can reproduce that. I suspect there is some element of your configuration that we don't have in OE-Core, for example, some kind of SRCREV set to AUTOREV that is causing process execution (git to get the revision) and that process execution is confusing things if the parse is interrupted at the wrong time.

To move forward with this we need a test case we can reproduce. Is there a way you could create a simpler setup we could reproduce this with?

Can you reproduce this just with OE-Core without other layers added?

> I think the originally reported bug here is fixed (bitbake -g hanging) but the report has changed to Ctrl+C interrupting a failing parse can hang?

In a bash infinite while-loop,
I repeated calling "bitbake -g", 
which would return "1" with "ERROR: ExpansionError during parsing /tmp/rickliu/stingray-bcm958802a8046c/poky/build/../meta-brcm/meta-datacenter/recipes-bsp/nitro/brcm-nitro.bb".

But this works (return 1 with ERROR) only for first several iteration,
and eventually the bash infinite while-loop will stop
and wait for "bitbake -g" command to return (bitbake -g starts to hang).


After "bitbake -g" hangs for a couple minutes,
then I start to try "Ctrl+C" to abort the "bitbake -g",
but this "Ctrl+C" abort would results stale bitbake processes,
which produces "unable to connect to bitbake server" errors.


I'll need to manually kill all of stale bitbake processes and remove *.lock files
to restore the machine back to normal state to start new bitbake commands.


So the original issue still exists: 
"Repeat of a failed bitbake -g eventually would cause bitbake hang"
I'll need to repeat 5~10 times of "bitbake" to reproduce the bitbake hang.


> I suspect there is some element of your configuration that we don't have in OE-Core, for example, some kind of SRCREV set to AUTOREV that is causing process execution (git to get the revision) and that process execution is confusing things if the parse is interrupted at the wrong time.

Yes, all of our recipes use SRC_URI with GIT repos
and use AUTOREV, *_SRCREV, SRCREV_FORMAT (for multiple SRC_URI with GIT repos).


> Can you reproduce this just with OE-Core without other layers added?
I'll try to see if I can find any OE-Core recipe is using SRC_URI with GIT repos
and uses AUTOREV, SRCREV, and SRCREV_FORMAT,
and try to reproduce the issue.

The originally reported bug has been fixed. I wrote several torture tests to try and reproduce the second problem, but was unable to see anything. If that issues occurs again, we can reopen (or open a new bug)