Bug 3887 - Merge buildstats into build history and have one data collection mechanism
Summary: Merge buildstats into build history and have one data collection mechanism
Status: RESOLVED WONTFIX
Alias: None
Product: Build Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 5.99
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: Future
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on: 5709
Blocks: 4683
  Show dependency tree
 
Reported: 2013-02-13 20:18 UTC by Laurentiu Serban
Modified: 2023-12-21 16:03 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Laurentiu Serban 2013-02-13 20:18:06 UTC
Merge buildstats into build history and have one data collection mechanism, provide a common dataset and a framework for extracting relevant data
Comment 1 Stoicescu Cornel 2013-05-03 06:36:18 UTC
This is the proposed implementation method:
-move buildstats in the buildhistory directory
-make buildstats compatible with git (rework directory structure)
-no merging will be done between buildstats and buildhistory information and directory structure; this way they will be able to maintain compatibility with related scrips like buildhistory-diff.
-when git is enabled, buildhistory's commit task will be performed upon both buildhistory and buildstats as it is triggered on bb.event.BuildCompleted . Buildstats and buildhistory should not submit any changes upon that event.

RP: Do you think the above implementation is appropriate? Also can you please provide me with a list of scripts that you are aware of related to buildhistory/buildstats? I need to look for any hard-coded links to <build_directory>/buildstats (where the build directory is acquired by variable but the buildstats directory is hard set in it). I will do my best to identify all of them though.

Regards,
Corneliu
Comment 2 Richard Purdie 2013-05-03 07:15:20 UTC
Paul is probably a better person to answer questions related to buildhistory data format
Comment 3 Paul Eggleton 2013-08-16 16:12:36 UTC
Sorry for the delay in replying. The above plan sounds OK; I'm still a bit concerned about moving buildstats under buildhistory by default though because of the amount of noise it will generate in the git logs (no problem if we want to configure it this way on the autobuilder by setting BUILDSTATS_BASE). Let's look at what changes will be needed to buildstats to fit within the git repository and go from there. 

As for what looks into buildhistory, scripts/buildhistory-diff and the buildhistory-web application both use meta/lib/oe/buildhistory_analysis.py to analyse the changes. There is also a scripts/buildhistory-collect-srcrevs script. Web Hob is also going to be collecting some information from the buildhistory repository, you should be able to see what there is so far of that here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/bitbake/lib/bb/ui/buildinfohelper.py?h=webhob-poky/master

(Although this is subject to change since Web Hob is still being developed.)
Comment 4 Stoicescu Cornel 2013-08-28 13:27:33 UTC
Here is the current state of this feature: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=cornel/bstats4

Changelog:

    Modify buildstats to be merged inside buildhistory

    - added buildstats inheritance inside buildhistory
    - reworked the buildstats directory structure not to contain the build name(read: the build start date). Now it only contains the target name and machine. This is usefull in order to reduce git noise.
    - because builds are no longer separated by build name(read: date) it is necessary to remake(remove and create from scratch) the buildstats folder for each build in order keep buildstats compatible with t


Working on improvements and testing.
Comment 6 Stoicescu Cornel 2013-11-01 17:10:46 UTC
patch sent for review on oe-core mailing list.
Comment 7 Stoicescu Cornel 2014-01-09 14:12:07 UTC
Because event handlers cannot be ordered, buildhistory_eventhandler is finished before buildstats_eventhandler and this results in some data being left out when buildhistory uses git to commit data.

Marking this feature as blocked by bug 5709
Comment 8 Stoicescu Cornel 2014-04-02 07:15:05 UTC
Moving to 1.7 . Still blocked by bug 5709 unless we go down a different implementation path.
Comment 9 Armin Kuster 2019-12-22 20:56:28 UTC
does result too do this?

I think so.

if not, please send patches
Comment 10 Randy MacLeod 2023-12-21 16:03:43 UTC
buildstats and build history do different things and we're not planning to merge them.