Bug 4151

Summary:	Improvments for the performance tests script
Product:	[QA/Testing] Build Testing	Reporter:	Stefan Stanacar <sstncr>
Component:	general	Assignee:	Stefan Stanacar <sstncr>
Status:	VERIFIED FIXED	QA Contact:
Severity:	enhancement
Priority:	Medium+	CC:	alexandru.c.georgescu, david.c.stewart, richard.purdie
Version:	unspecified
Target Milestone:	1.5 M5
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know
Bug Depends on:
Bug Blocks:	4726

Description Stefan Stanacar 2013-03-29 12:28:15 UTC

The current performance tests script which is used to generate the data in https://wiki.yoctoproject.org/wiki/Performance_Test needs some improvement.
We need to:

a) Save all the data to a common log file in something like csv format
b) Have a way of handling multiple runs of the same revision (gives us
   standard deviation data)
c) Allow only specific performance tests to be run (e.g. when adding a
   new test, able to run it over the previous revisions)
d) Look at tagging specific points in history with events (like gtk+
   -native dependency dropped)
e) Support multiple machines in the file format so we can centrally
   collect data
f) In the logging mechanism allow saving of "fixup" data, which
   corrections were applied to the revision
g) Mark other "master" revisions as being 1.4M5+gitA, +gitB etc. so we
   can place random revisions into sequence between the milestones.
h) Actually get the script merged into master

It would also be nice to have:
 - a way to generate graphs from the data (probably using another tool/script which interprets the output/data file from the first script)
 - run these tests more regulary than once a milestone (maybe weekly). This can be done on a dedicated machine for this task, but also, there is no technical reason why it can't also run as part of AB builds once a week. 


Note: Point a) (and some b)) is partially done right now, we just need to reconsider it a bit with the rest of the points:

$ cat build-perf-test/globalres.log 
98292d1ef1cb44cf96f0a28fbca6439eda55ce1b
1:16:23,16:37.16,1:20:09,17:04.16
45f95b5f3381097672dd43077f267aa716a02b4c
1:11:29,14:53.02,1:13:36,9:55.93,0:45.26,0:15.31,0:01.92
98292d1ef1cb44cf96f0a28fbca6439eda55ce1b
1:12:20,15:06.17,1:16:32,10:27.73,0:45.90,0:15.21,0:01.98
7d22ef28e64de06b6238d0ef9b664c0bee6efa46
1:08:56,12:31.16,1:08:36,5:25.14,0:27.62,0:16.09,0:01.90
d823759b4594143d522eae0b2a2498436a6dcb1e
1:03:48,12:26.28,1:05:24,5:30.01,0:15.69,0:15.99,0:02.86
bdca49274173153b1b0a72887de6321b4756b684
1:05:48,11:03.47,1:06:04,5:31.13,0:29.69,0:17.26,0:02.04

Comment 1 Stefan Stanacar 2013-05-09 10:11:06 UTC

Just updating the bug with the latest rev:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=735656bf6290d38be00cc0bd31e3c86f9a398e2f

The results file looks like this and should cover points a), b) and e):

$ cat build-perf-test/globalres.log 
thunderbird,3e0455f3cf104ce95f40aa989b21d9a70efee34c,1:08:07,13:23.60,1:08:30,5:27.13,0:29.31,0:16.93,0:02.00
thunderbird,3e0455f3cf104ce95f40aa989b21d9a70efee34c,1:08:23,13:53.54,1:08:48,5:30.45,0:30.23,0:17.31,0:02.01
thunderbird,d6cc4edb533b196753cf90820d1e2b9ccd8fed7c,1:18:16,16:08.98,1:19:56,16:29.13,0:45.07,0:15.25,0:01.92

Comment 2 Alexandru Georgescu 2013-08-26 07:20:26 UTC

Hi Stefan,

I have two more request from the team, regarding Performance test.

Please change the size displayed from GB to MB in order to have a more accurate track for the size used on disk.

Comment 3 Alexandru Georgescu 2013-08-26 07:24:39 UTC

The second one is to have another wiki where to store all the data we gather.

Currently we run multiple times the script on the same commit to be able to make an average and overtime the wiki gets bigger and hard to browse. The other wiki should have all this data stored.

Comment 4 Stefan Stanacar 2013-10-01 12:32:59 UTC

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/contrib/build-perf-test.sh?id=b7308e7814a9a3c7dee2477acca7e4b7a176cc1d
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/contrib/build-perf-test.sh?id=30d3c8d11e0c81a22a422346fe8c11ef37f2f33f

AlexG first comment: size has been changed to KB
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/contrib/build-perf-test.sh?id=576a19ed6c71ea04b39df806d4838cc74a015b26

AlexG second comment: - the wiki page has been clean-up a bit, but I'm not gonna delete results or do averages (we never did) no matter how hard is to browse (doing time averages is not something nice in statistics, plus without a standard deviation they are meaningless).

My first comment: as it is now it covers points a), b), e), g), h)
So c) d) and f) as improvements aren't gonna make it and honestly I don't have any idea how to do them in the current form of the script. It was my bad idea to have entire list in one bug.
I'm going to close this as it is, if anyone has any objections please feel free to reopen or open new bugs for specific issues.

fedora19,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:12:34,10:09.63,1:10:30,4:03.92,0:33.53,0:17.84,0:01.92,28713992,4835460
fedora19,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:16:37,11:06.88,1:12:16,4:09.17,0:33.86,0:17.71,0:01.92,25771504,4858908
fedora19,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:19:37,10:38.18,1:12:10,4:03.70,0:34.59,0:18.54,0:02.23,25801440,4887964
fedora19,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:20:02,11:43.17,1:13:52,3:58.63,0:35.54,0:19.14,0:02.13,25704568,4911492
fedora19,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:19:42,11:19.27,1:15:23,3:55.77,0:36.15,0:19.22,0:02.25,25786336,4921400

ubuntu12,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:18:19,13:41.56,1:11:33,4:53.58,0:28.27,0:15.41,0:01.54,25688600,4898748
ubuntu12,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:22:17,14:01.95,1:11:01,4:54.03,0:28.51,0:15.20,0:01.55,25748224,4897116
ubuntu12,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:14:39,14:14.73,1:11:21,4:53.20,0:25.47,0:12.80,0:01.48,25770168,4924972
ubuntu12,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:17:49,14:02.55,1:12:09,4:51.49,0:25.57,0:12.96,0:01.60,25760572,4915828

Comment 5 Stefan Stanacar 2013-10-01 12:36:18 UTC

In my previous comment output has been wrapped it should look nicer, one run per log line. 

fedora19,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:19:37,10:38.18,1:12:10,4:03.70,0:34.59,0:18.54,0:02.23,25801440,4887964
fedora19,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:19:42,11:19.27,1:15:23,3:55.77,0:36.15,0:19.22,0:02.25,25786336,4921400
fedora19,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:23:20,12:42.28,1:16:41,4:08.14,0:35.09,0:18.98,0:02.29,26535116,5049040