10330 2016-09-26 16:32:49 +0000 bitbake with memory resident don't report SSTATE progress 2016-10-05 15:08:41 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED YP 2.2 M4 RC1 BUILD BLOCKING BUG Medium+ normal 2.2 M4 1 anibal.limon anibal.limon anibal.limon bluelightning poky.bs.watcher poky.watcher richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 66592 0 anibal.limon 2016-09-26 16:32:49 +0000 When try to use bitbake with memory resident and SSTATE_MIRRORs the sstate check object availability bar gets stalled at 3% and the build continues with background processes executing, For reproduce this bug you need to config SSTATE_MIRRORs in local.conf. $ source oe-init-build-env-memres $ bitbake core-image-full-cmdline Loading cache: 100% |############################################| Time: 0:00:01 Loaded 1316 entries from dependency cache. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.31.1" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "aarch64-poky-linux" MACHINE = "qemuarm64" DISTRO = "poky" DISTRO_VERSION = "2.1+snapshot-20160926" TUNE_FEATURES = "aarch64" TARGET_FPU = "" meta meta-poky meta-yocto-bsp = "alimon/simple_patches:725e66e1d08ae000d8f68455ddca0e192080dc1f" Initialising tasks: 100% |########################################################| Time: 0:00:06 Checking sstate mirror object availability: 1% | | ETA: 0:00:01 66648 1 leonardo.sandoval.gonzalez 2016-09-27 19:09:33 +0000 I tried twice launching the proposed commmands, first time it get stucks on 0% second time on 8%. One more thing: The issue is not observed with oe-init-build-env setup script so only affecting memres. 66802 2 jose.a.lamego 2016-10-03 14:21:19 +0000 This seems to be a concurrency related problem: Progress bar behaves as expected if the fetch tests are forced to be run in a single thread (nproc = 1). Need to find the relation between this behavior and the memory resident setting. 66855 3 richard.purdie 2016-10-04 09:44:26 +0000 I believe Anibal has made progress and this looks like a thread concurrency issue in bb.event.fire() and is related to how sstate is using threading. 66860 4 anibal.limon 2016-10-04 13:27:26 +0000 (In reply to comment #3) > I believe Anibal has made progress and this looks like a thread concurrency > issue in bb.event.fire() and is related to how sstate is using threading. Yes, i found that events are generated into the event.fire by the server but the client (UI) don't receive the ProgressEnd event. So the next patch fixes the issue but we need to discuss how to address it. @@ -145,12 +145,17 @@ def print_ui_queue(): if isinstance(event, logging.LogRecord): logger.handle(event) +import threading +ui_fire_lock = threading.Lock() + def fire_ui_handlers(event, d): if not _uiready: # No UI handlers registered yet, queue up the messages ui_queue.append(event) return + ui_fire_lock.acquire() + errors = [] for h in _ui_handlers: #print "Sending event %s" % event @@ -169,6 +174,9 @@ def fire_ui_handlers(event, d): for h in errors: del _ui_handlers[h] + + ui_fire_lock.release() + def fire(event, d): """Fire off an Event""" @@ -176,7 +184,6 @@ def fire(event, d): # desired so they get the task based datastore. # UI handlers need to be fired in the server context so we defer this. They # don't have a datastore so the datastore context isn't a problem. - fire_class_handlers(event, d) if worker_fire: worker_fire(event, d) 66874 5 anibal.limon 2016-10-04 16:47:16 +0000 Patches sent at bitbake devel an oe-core, http://lists.openembedded.org/pipermail/bitbake-devel/2016-October/008129.html http://lists.openembedded.org/pipermail/openembedded-core/2016-October/127274.html 66942 6 anibal.limon 2016-10-05 15:08:41 +0000 At bitbake, http://git.openembedded.org/bitbake/commit/?id=a583dc0b296415ec904c081c4de96ceef46732a8 At poky, http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=f2053844958325496a9387874a8f3182400b71ca http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=79012f943d3b3af1f439022f05a1287a383cd3f8