The current performance tests script which is used to generate the data in https://wiki.yoctoproject.org/wiki/Performance_Test needs some improvement. We need to: a) Save all the data to a common log file in something like csv format b) Have a way of handling multiple runs of the same revision (gives us standard deviation data) c) Allow only specific performance tests to be run (e.g. when adding a new test, able to run it over the previous revisions) d) Look at tagging specific points in history with events (like gtk+ -native dependency dropped) e) Support multiple machines in the file format so we can centrally collect data f) In the logging mechanism allow saving of "fixup" data, which corrections were applied to the revision g) Mark other "master" revisions as being 1.4M5+gitA, +gitB etc. so we can place random revisions into sequence between the milestones. h) Actually get the script merged into master It would also be nice to have: - a way to generate graphs from the data (probably using another tool/script which interprets the output/data file from the first script) - run these tests more regulary than once a milestone (maybe weekly). This can be done on a dedicated machine for this task, but also, there is no technical reason why it can't also run as part of AB builds once a week. Note: Point a) (and some b)) is partially done right now, we just need to reconsider it a bit with the rest of the points: $ cat build-perf-test/globalres.log 98292d1ef1cb44cf96f0a28fbca6439eda55ce1b 1:16:23,16:37.16,1:20:09,17:04.16 45f95b5f3381097672dd43077f267aa716a02b4c 1:11:29,14:53.02,1:13:36,9:55.93,0:45.26,0:15.31,0:01.92 98292d1ef1cb44cf96f0a28fbca6439eda55ce1b 1:12:20,15:06.17,1:16:32,10:27.73,0:45.90,0:15.21,0:01.98 7d22ef28e64de06b6238d0ef9b664c0bee6efa46 1:08:56,12:31.16,1:08:36,5:25.14,0:27.62,0:16.09,0:01.90 d823759b4594143d522eae0b2a2498436a6dcb1e 1:03:48,12:26.28,1:05:24,5:30.01,0:15.69,0:15.99,0:02.86 bdca49274173153b1b0a72887de6321b4756b684 1:05:48,11:03.47,1:06:04,5:31.13,0:29.69,0:17.26,0:02.04
Just updating the bug with the latest rev: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=735656bf6290d38be00cc0bd31e3c86f9a398e2f The results file looks like this and should cover points a), b) and e): $ cat build-perf-test/globalres.log thunderbird,3e0455f3cf104ce95f40aa989b21d9a70efee34c,1:08:07,13:23.60,1:08:30,5:27.13,0:29.31,0:16.93,0:02.00 thunderbird,3e0455f3cf104ce95f40aa989b21d9a70efee34c,1:08:23,13:53.54,1:08:48,5:30.45,0:30.23,0:17.31,0:02.01 thunderbird,d6cc4edb533b196753cf90820d1e2b9ccd8fed7c,1:18:16,16:08.98,1:19:56,16:29.13,0:45.07,0:15.25,0:01.92
Hi Stefan, I have two more request from the team, regarding Performance test. Please change the size displayed from GB to MB in order to have a more accurate track for the size used on disk.
The second one is to have another wiki where to store all the data we gather. Currently we run multiple times the script on the same commit to be able to make an average and overtime the wiki gets bigger and hard to browse. The other wiki should have all this data stored.
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/contrib/build-perf-test.sh?id=b7308e7814a9a3c7dee2477acca7e4b7a176cc1d http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/contrib/build-perf-test.sh?id=30d3c8d11e0c81a22a422346fe8c11ef37f2f33f AlexG first comment: size has been changed to KB http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/contrib/build-perf-test.sh?id=576a19ed6c71ea04b39df806d4838cc74a015b26 AlexG second comment: - the wiki page has been clean-up a bit, but I'm not gonna delete results or do averages (we never did) no matter how hard is to browse (doing time averages is not something nice in statistics, plus without a standard deviation they are meaningless). My first comment: as it is now it covers points a), b), e), g), h) So c) d) and f) as improvements aren't gonna make it and honestly I don't have any idea how to do them in the current form of the script. It was my bad idea to have entire list in one bug. I'm going to close this as it is, if anyone has any objections please feel free to reopen or open new bugs for specific issues. fedora19,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:12:34,10:09.63,1:10:30,4:03.92,0:33.53,0:17.84,0:01.92,28713992,4835460 fedora19,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:16:37,11:06.88,1:12:16,4:09.17,0:33.86,0:17.71,0:01.92,25771504,4858908 fedora19,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:19:37,10:38.18,1:12:10,4:03.70,0:34.59,0:18.54,0:02.23,25801440,4887964 fedora19,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:20:02,11:43.17,1:13:52,3:58.63,0:35.54,0:19.14,0:02.13,25704568,4911492 fedora19,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:19:42,11:19.27,1:15:23,3:55.77,0:36.15,0:19.22,0:02.25,25786336,4921400 ubuntu12,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:18:19,13:41.56,1:11:33,4:53.58,0:28.27,0:15.41,0:01.54,25688600,4898748 ubuntu12,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:22:17,14:01.95,1:11:01,4:54.03,0:28.51,0:15.20,0:01.55,25748224,4897116 ubuntu12,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:14:39,14:14.73,1:11:21,4:53.20,0:25.47,0:12.80,0:01.48,25770168,4924972 ubuntu12,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:17:49,14:02.55,1:12:09,4:51.49,0:25.57,0:12.96,0:01.60,25760572,4915828
In my previous comment output has been wrapped it should look nicer, one run per log line. fedora19,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:19:37,10:38.18,1:12:10,4:03.70,0:34.59,0:18.54,0:02.23,25801440,4887964 fedora19,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:19:42,11:19.27,1:15:23,3:55.77,0:36.15,0:19.22,0:02.25,25786336,4921400 fedora19,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:23:20,12:42.28,1:16:41,4:08.14,0:35.09,0:18.98,0:02.29,26535116,5049040