Bug 3886

Summary: Script to automatically run the performance tests and consolidates the data
Product: [QA/Testing] Build Testing Reporter: Laurentiu Serban <laurentiu.serban>
Component: generalAssignee: Stefan Stanacar <sstncr>
Status: VERIFIED FIXED QA Contact:
Severity: enhancement    
Priority: High CC: corneliux.stoicescu, sgw
Version: 1.4   
Target Milestone: 1.4 M5   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Laurentiu Serban 2013-02-13 20:12:09 UTC 
Write a script which performs the following tests then collects up up the appropriate data.

Overall Build Time
Prerequisite: Downloaded Sources, no sstate
Measurement: Wall clock of "bitbake core-image-sato"

Image build from SState
Prerequisite: Have already populated full sstate cache
Measurement: Wall clock of "bitbake core-image-sato -c rootfs"

Time to build the kernel
Prerequisite: Have existing build of kernel, run "bitbake virtual/kernel -c clean"
Measurement: Wall clock of "bitbake virtual/kernel"

Disk space usage of a build (no rm_work)
Prerequisite: Downloaded Sources, no sstate
Measurement: Size of tmp directory after a "bitbake c-re-image-sato"

Disk space usage of a build with rm_work 
Prerequisite: Downloaded Sources, no sstate
Measurement: Size of tmp directory after a "bitbake c-re-image-sato" but with rm_work enabled
Comment 2 Stefan Stanacar 2013-02-21 14:04:34 UTC 
Oki, latest version of the script should be here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/scripts/contrib/build-perf-test.sh?h=stefans/perf-test

It has two outputs, one global results file, which looks smth like:
7d22ef28e64de06b6238d0ef9b664c0bee6efa46
1:07:16,12:48.31,1:11:25,5:14.22

and a more verbose one per commit.

I've did two runs on the same system (to ensure reproductiblity), and published some results here: 
https://wiki.yoctoproject.org/wiki/Performance_Test#1.4_Perfomance_Data

What still needs to be done: 
  - improve output
  - gather some data from buildstats
  - run the script on every milestone from now on, on the standard Ubuntu machine (I had some problems with mine, that's why on the wiki page you'll see a Fedora system used).
Comment 3 Stoicescu Cornel 2013-03-22 09:19:29 UTC 
The script works as intended. 

QA uses it in full pass tests to evaluate build performance. The results can be viewed here: https://wiki.yoctoproject.org/wiki/Performance_Test
Comment 4 Stefan Stanacar 2013-04-03 16:12:28 UTC 
For record keeping purposes: this has been merged in master, so that link to poky-contrib is obsolete: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=851f1e368bea57e090a924330a08a793abfcfa55
It can be found under scripts/contrib folder. I've updated with the new location on the wiki page too.