Bug 10330 - bitbake with memory resident don't report SSTATE progress
Summary: bitbake with memory resident don't report SSTATE progress
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.2 M4
Assignee: Aníbal Limón
QA Contact:
URL:
Whiteboard: YP 2.2 M4 RC1 BUILD BLOCKING BUG
Depends on:
Blocks:
 
Reported: 2016-09-26 16:32 UTC by Aníbal Limón
Modified: 2016-10-05 15:08 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Aníbal Limón 2016-09-26 16:32:49 UTC 
When try to use bitbake with memory resident and SSTATE_MIRRORs the sstate check object availability bar gets stalled at 3% and the build continues with background processes executing,

For reproduce this bug you need to config SSTATE_MIRRORs in local.conf.

$ source oe-init-build-env-memres
$ bitbake core-image-full-cmdline
Loading cache: 100% |############################################| Time: 0:00:01
Loaded 1316 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.31.1"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS        = "aarch64-poky-linux"
MACHINE           = "qemuarm64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.1+snapshot-20160926"
TUNE_FEATURES     = "aarch64"
TARGET_FPU        = ""
meta              
meta-poky         
meta-yocto-bsp    = "alimon/simple_patches:725e66e1d08ae000d8f68455ddca0e192080dc1f"

Initialising tasks: 100% |########################################################| Time: 0:00:06
Checking sstate mirror object availability:   1% |                                | ETA:  0:00:01
Comment 1 Leonardo Sandoval Gonzalez 2016-09-27 19:09:33 UTC 
I tried twice launching the proposed commmands, first time it get stucks on 0% second time on 8%. One more thing: The issue is not observed with oe-init-build-env setup script so only affecting memres.
Comment 2 Jose Lamego 2016-10-03 14:21:19 UTC 
This seems to be a concurrency related problem: Progress bar behaves as expected if the fetch tests are forced to be run in a single thread (nproc = 1).
Need to find the relation between this behavior and the memory resident setting.
Comment 3 Richard Purdie 2016-10-04 09:44:26 UTC 
I believe Anibal has made progress and this looks like a thread concurrency issue in bb.event.fire() and is related to how sstate is using threading.
Comment 4 Aníbal Limón 2016-10-04 13:27:26 UTC 
(In reply to comment #3)
> I believe Anibal has made progress and this looks like a thread concurrency
> issue in bb.event.fire() and is related to how sstate is using threading.

Yes, i found that events are generated into the event.fire by the server but the client (UI) don't receive the ProgressEnd event. So the next patch fixes the issue but we need to discuss how to address it.

@@ -145,12 +145,17 @@ def print_ui_queue():
             if isinstance(event, logging.LogRecord):
                 logger.handle(event)
 
+import threading
+ui_fire_lock = threading.Lock()
+
 def fire_ui_handlers(event, d):
     if not _uiready:
         # No UI handlers registered yet, queue up the messages
         ui_queue.append(event)
         return
 
+    ui_fire_lock.acquire()
+
     errors = []
     for h in _ui_handlers:
         #print "Sending event %s" % event
@@ -169,6 +174,9 @@ def fire_ui_handlers(event, d):
     for h in errors:
         del _ui_handlers[h]
 
+
+    ui_fire_lock.release()
+
 def fire(event, d):
     """Fire off an Event"""
 
@@ -176,7 +184,6 @@ def fire(event, d):
     # desired so they get the task based datastore.
     # UI handlers need to be fired in the server context so we defer this. They
     # don't have a datastore so the datastore context isn't a problem.
-
     fire_class_handlers(event, d)
     if worker_fire:
         worker_fire(event, d)