Bug 9971

Summary:	Toaster does not report progress while setscene tasks are running
Product:	[Build System, Metadata & Runtime] Toaster	Reporter:	Belen Barros Pena <belen.barros.pena>
Component:	toaster	Assignee:	David Reyna <david.reyna>
Status:	RESOLVED FIXED	QA Contact:	Libertad <libertad.gonzalez.de.la.cruz>
Severity:	normal
Priority:	Medium	CC:	belen.barros.pena, brian.avery, david.reyna, jessica.zhang, stanciux.mihail
Version:	2.2
Target Milestone:	2.4 M2
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)
Bug Depends on:	9631
Bug Blocks:

Description Belen Barros Pena 2016-07-18 15:04:47 UTC

The best way to see the problem is building from the command line. Here are the steps to reproduce:

1. Start Toaster, and from the build directory, start a build from the command line (for example, core-image-minimal)

2. When the build completes, remote the /build/tmp directory

3. Then, from the command line, run again the same build

Outcome: in your terminal you will see the setscene tasks running, while in Toaster you will see an empty progress bar with 0% tasks completed. The progress will stay at 0% until non-setscene tasks start, at which point Toaster will start reporting task progress.

Comment 1 brian avery 2016-07-21 05:57:43 UTC

joshua did some work on toaster.bbclass commit f5033175ef51cf234ab5a85293a30bd9d7440d75 which changed (improved) some of the postfuncs for the sstate setscene tasks.  This note is mostly a reminder about that work so we can look at it when we get to this bug in case it's related.

Comment 2 David Reyna 2017-07-01 01:52:12 UTC

I have investigated and I have found the issue and resolution.

The way task progress works now is:
(a) The 'bb.event.DepTreeGenerated' event passes each task as it is processed, and Toaster adds it to build's full task list.
(b) Task started events get passed to "store_started_task()" in "buildinfohelper.py", which increments the respective "order" value so that the tasks are easy ordered in the Task View page.
(c) The task progress is calculated in "models.py>completeper()" as the number of tasks that have an order value versus the total number of tasks.

There are three problems with this implementation.
1. Setscene tasks explicitly do not get an "order" value. This was because we did not want these tasks to appear in the Task View page, but that means they do not get counted as per this defect.
2. The "order" value is set when the task is started instead of when it is completed, so the values are incorrectly optimistic.
3. The computation (while saving database space) is very inefficient, with the continual lookup and filtering of thousands of task records per progress update.

The fix for for this defect (issues #1 and #2) is to simply count the tasks with a completion status instead of those with an order value.

For completeness, the fix for #3 update these values incrementally with no look-ups, and additionally move all such transient values out of the models and into a runtime build context like "localhostbecontroller.py", and will be handled in a separate defect.

Comment 3 David Reyna 2017-08-07 19:25:36 UTC

commit 36568b9e1e079f55a6d2e7bad0fd3ca27c90bb53