Bug 5300

Summary: Modify buildstats implementation to use the bb.build.Task* timestamp
Product: [Build System, Metadata & Runtime] Toaster Reporter: Belen Barros Pena <belen.barros.pena>
Component: toasterAssignee: Marius Avram <marius.avram>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: alexandru.damian, belen.barros.pena, elizabeth.flanagan, marius.avram, sgw, Yuan.Sun2
Version: unspecified   
Target Milestone: 1.6 M5   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 5485    
Bug Blocks:    
Attachments:
Description Flags
sysvinit-inittab.do_package info from build stats
none
sysvinit-inittab.do_package info from web hob none

Description Belen Barros Pena 2013-10-01 14:10:00 UTC
Created attachment 1537 [details]
sysvinit-inittab.do_package info from build stats

We are now collecting tasks cpu usage and disk i/o from build stats. However, the task time reported by Web Hob doesn't seem to match the build stats information (see attached files).

How is the task time being obtained and why is it different from the information reported by build stats?

webhob-master-1_5-2609
Comment 1 Belen Barros Pena 2013-10-01 14:10:44 UTC
Created attachment 1538 [details]
sysvinit-inittab.do_package info from web hob
Comment 2 Alexandru Damian 2013-10-01 15:54:07 UTC
Tentative patch for both 5300 and 5299 in testing.

The root cause for both bugs seems that the native- and normal tasks had a naming collision in the state tracking structure, and the tasks may got overwritten.
Comment 3 Alexandru Damian 2013-10-01 15:56:38 UTC
Fix available on adamian/webhob-minimal-1_5-3
Comment 4 Belen Barros Pena 2013-10-03 10:17:43 UTC
Still not working for me in 

adamian/webhob-minimal-1_5-3

do_package.eglibc-initial

elapsed time in web hob: 0.018142
elapsed time in build stats: 0.57
Comment 5 Alexandru Damian 2013-10-03 15:18:27 UTC
IMHO, this happens because on buildstats.class takes the completion time after creating the buildstats directory hierarchy and opening the destination file, which can take a significant amount of time on a loaded system (see write_task_data()).

webhob takes the completion time as soon as the event is received, so it's more accurate.
Comment 6 Belen Barros Pena 2013-10-11 10:11:21 UTC
If this means that web hob times are the correct ones, should we do something about build stats?

I am concerned about our different tools reporting different information: it makes the lot look unreliable. But I am not sure if build stats can be made to work in any other way.
Comment 7 Alexandru Damian 2013-10-21 10:17:50 UTC
Tentatively assigned to Beth, should we want to change the buildstats info.
Comment 8 Alexandru Damian 2013-12-30 12:38:20 UTC
Moved to new, Beth, can you please let us know when you'll take a look at this ?
Comment 9 Marius Avram 2014-02-19 10:04:20 UTC
This bug is very similar to:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5485 , which is in process of solving.
Comment 10 Alexandru Damian 2014-02-19 18:46:29 UTC
We are adding time-stamps to the bb.build.Task* events. This will allow time-based measurement of task duration synchronously during task execution, thereby eliminating measurement errors based on time of processing events.

Marius, can you please modify the buildstats code as well ?
Comment 11 Marius Avram 2014-03-25 13:27:32 UTC
Solution merged to master:poky, commit: ba6caa021fea2edc5483bcde6d71122f2a2a871a
Comment 12 Yuan Sun 2014-04-08 08:06:23 UTC
Verified in 3e528d3e4bc542ca2f654c54363c56af0fb9f61b.