Bug 10330

Summary: bitbake with memory resident don't report SSTATE progress
Product: [Build System, Metadata & Runtime] BitBake Reporter: Aníbal Limón <anibal.limon>
Component: bitbakeAssignee: Aníbal Limón <anibal.limon>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: anibal.limon, bluelightning, poky.bs.watcher, poky.watcher, richard.purdie
Version: unspecified   
Target Milestone: 2.2 M4   
Hardware: x86   
OS: Multiple   
Whiteboard: YP 2.2 M4 RC1 BUILD BLOCKING BUG
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Aníbal Limón 2016-09-26 16:32:49 UTC 
When try to use bitbake with memory resident and SSTATE_MIRRORs the sstate check object availability bar gets stalled at 3% and the build continues with background processes executing,

For reproduce this bug you need to config SSTATE_MIRRORs in local.conf.

$ source oe-init-build-env-memres
$ bitbake core-image-full-cmdline
Loading cache: 100% |############################################| Time: 0:00:01
Loaded 1316 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.31.1"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS        = "aarch64-poky-linux"
MACHINE           = "qemuarm64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.1+snapshot-20160926"
TUNE_FEATURES     = "aarch64"
TARGET_FPU        = ""
meta              
meta-poky         
meta-yocto-bsp    = "alimon/simple_patches:725e66e1d08ae000d8f68455ddca0e192080dc1f"

Initialising tasks: 100% |########################################################| Time: 0:00:06
Checking sstate mirror object availability:   1% |                                | ETA:  0:00:01
Comment 1 Leonardo Sandoval Gonzalez 2016-09-27 19:09:33 UTC 
I tried twice launching the proposed commmands, first time it get stucks on 0% second time on 8%. One more thing: The issue is not observed with oe-init-build-env setup script so only affecting memres.
Comment 2 Jose Lamego 2016-10-03 14:21:19 UTC 
This seems to be a concurrency related problem: Progress bar behaves as expected if the fetch tests are forced to be run in a single thread (nproc = 1).
Need to find the relation between this behavior and the memory resident setting.
Comment 3 Richard Purdie 2016-10-04 09:44:26 UTC 
I believe Anibal has made progress and this looks like a thread concurrency issue in bb.event.fire() and is related to how sstate is using threading.
Comment 4 Aníbal Limón 2016-10-04 13:27:26 UTC 
(In reply to comment #3)
> I believe Anibal has made progress and this looks like a thread concurrency
> issue in bb.event.fire() and is related to how sstate is using threading.

Yes, i found that events are generated into the event.fire by the server but the client (UI) don't receive the ProgressEnd event. So the next patch fixes the issue but we need to discuss how to address it.

@@ -145,12 +145,17 @@ def print_ui_queue():
             if isinstance(event, logging.LogRecord):
                 logger.handle(event)
 
+import threading
+ui_fire_lock = threading.Lock()
+
 def fire_ui_handlers(event, d):
     if not _uiready:
         # No UI handlers registered yet, queue up the messages
         ui_queue.append(event)
         return
 
+    ui_fire_lock.acquire()
+
     errors = []
     for h in _ui_handlers:
         #print "Sending event %s" % event
@@ -169,6 +174,9 @@ def fire_ui_handlers(event, d):
     for h in errors:
         del _ui_handlers[h]
 
+
+    ui_fire_lock.release()
+
 def fire(event, d):
     """Fire off an Event"""
 
@@ -176,7 +184,6 @@ def fire(event, d):
     # desired so they get the task based datastore.
     # UI handlers need to be fired in the server context so we defer this. They
     # don't have a datastore so the datastore context isn't a problem.
-
     fire_class_handlers(event, d)
     if worker_fire:
         worker_fire(event, d)