Bug 10676

Summary: cooker: event queue overflowed
Product: [Build System, Metadata & Runtime] BitBake Reporter: Patrick Ohly <patrick.ohly>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact: David Lopez Barriba <davidx.lopez.barriba>
Severity: normal    
Priority: Medium+ CC: jose.perez.carranza, poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: 2.3 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Patrick Ohly 2016-11-16 08:19:54 UTC 
When producing a lot of output from shell commands which run in parallel (see bug #10668) as part of a task, I ended up with this failure:

ERROR: Running idle function
Traceback (most recent call last):
  File "/work/ostro-os/bitbake/lib/bb/server/process.py", line 142, in ProcessServer.idle_commands(delay=0.1, fds=[<multiprocessing.connection.Connection object at 0x7f67d0fe65f8>, <multiprocessing.connection.Connection object at 0x7f67d0b37080>]):
                 try:
    >                retval = function(self, data, False)
                     if retval is False:
  File "/work/ostro-os/bitbake/lib/bb/cooker.py", line 211, in _process_inotify_updates(server=<ProcessServer(ProcessServer-1, started)>, notifier_list=[<pyinotify.Notifier object at 0x7f67d0b3d6a0>, <pyinotify.Notifier object at 0x7f67d0b3d7f0>], abort=False):
                         n.read_events()
    >                    n.process_events()
                 return 1.0
  File "/work/ostro-os/bitbake/lib/pyinotify.py", line 1275, in Notifier.process_events():
                 else:
    >                self._default_proc_fun(revent)
             self._sys_proc_fun.cleanup()  # remove olds MOVED_* events records
  File "/work/ostro-os/bitbake/lib/bb/cooker.py", line 262, in BBCooker.notifications(event=<Event mask=0x4000 maskname=IN_Q_OVERFLOW >):
         def notifications(self, event):
    >        if not event.pathname in self.inotify_modified_files:
                 self.inotify_modified_files.append(event.pathname)
AttributeError: 'Event' object has no attribute 'pathname'

IN_Q_OVERFLOW is "Event queue overflowed". It seems reasonable that such an event doesn't have a pathname.

Not sure what bitbake can do in this case. At least it shouldn't crash as above.
Comment 1 Patrick Ohly 2016-11-16 09:05:43 UTC 
Handling a shell command like
 echo foo | sed -u 's/^/bar: /' & wait $!
required a slightly more complex workaround. In that case, the offending token is a list of three tuples.

Here's the updated workaround which handles that. Changing indention makes it fairly large. The basic idea is that if a token is a list, process_token_list will be called recursively for the members of that list.

diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py
index 41484ec..8673b77 100644
--- a/bitbake/lib/bb/codeparser.py
+++ b/bitbake/lib/bb/codeparser.py
@@ -389,25 +389,24 @@ class ShellParser():
             "case_clause": case_clause,
         }
 
-        for token in tokens:
-            try:
+        def process_token_list(tokens):
+            for token in tokens:
+                if isinstance(token, list):
+                    process_token_list(token)
+                    continue
                 name, value = token
-            except ValueError:
-                if isinstance(token, list) and len(token) == 1:
-                    bb.warn('Unexpected token (need list or tuple with two entries): %s' % token)
-                    name, value = token[0]
-                else:
-                    bb.fatal('Cannot split %s into name and value (need list or tuple with two entries)' % token)
-            try:
-                more_tokens, words = token_handlers[name](value)
-            except KeyError:
-                raise NotImplementedError("Unsupported token type " + name)
+                try:
+                    more_tokens, words = token_handlers[name](value)
+                except KeyError:
+                    raise NotImplementedError("Unsupported token type " + name)
+
+                if more_tokens:
+                    self.process_tokens(more_tokens)
 
-            if more_tokens:
-                self.process_tokens(more_tokens)
+                if words:
+                    self.process_words(words)
 
-            if words:
-                self.process_words(words)
+        process_token_list(tokens)
 
     def process_words(self, words):
         """Process a set of 'words' in pyshyacc parlance, which includes
Comment 2 Patrick Ohly 2016-11-16 09:06:11 UTC 
Oops, wrong bug. Please ignore comment #1.