5049 2013-08-26 16:04:30 +0000 Create a WDD like charts for the Performance Test script that we are running regularly 2017-05-23 07:51:47 +0000 1 1 10 QA/Testing Build Testing general unspecified x86 Multiple RESOLVED FIXED TRT Medium+ normal 2.4 M1 6319 1 alexandru.c.georgescu markus.lehtonen benjamin.esquivel jefro joshuagloe markus.lehtonen melissa mhalstead richard.purdie sgw sstncr --- --- Don't know oldest_to_newest 35922 0 1442 alexandru.c.georgescu 2013-08-26 16:04:30 +0000 Created attachment 1442 Sample CSV file. Since late 1.4 we started to run the performance_test script https://bugzilla.yoctoproject.org/show_bug.cgi?id=4151 which basically measures build time for the images or kernel with different options. The data is gathered for every milestone build but also more often, for the 1.5 builds. This information is available in the Performance Test wiki: https://wiki.yoctoproject.org/wiki/Performance_Test but is in table format. We want to have an easier way to display it, using charts (https://developers.google.com/chart/interactive/docs/gallery) like the ones used for displaying WDD information: https://wiki.yoctoproject.org/charts/combo.html to track the build performance enhancements or to prevent regressions. Some of the requirements would be: - have a separate chart that displayes the performance for the following tests: bitbake core-image-sato, bitbake, virtual/kernel, bitbake, core-image-sato (rm_work), bitbake core-image-sato -c rootfs, bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/), bitbake -p, size of tmp dir, size of tmp dir (rm_work). - the tests will be run on two machines for the same commit on different distributions. Every chart should have the information for the both machines so that they can be easily compared for a specific commit. - the charts will be updated from a CSV file, containing all the information needed to populate them. - for the build time charts, there should be a way to be able to distinct the two hosts (ubuntu and thunderbird). Their names will be on the CSV - for the build time charts, the Oy axis must represent the time (available in CSV file) - for the build time and disk size charts, the Oy axis must represent the commit or the git describe (all available in the CSV file) - for the disk size charts, the Oy axis will represent the size (in MB - note that the CSV file is in GB, but the script will be updated) Sample CSV file, attached. Expect more requirements in the future. 36495 1 mhalstead 2013-09-10 16:19:29 +0000 An example of the first chart is up at https://wiki.yoctoproject.org/charts/timing/timing2.html showing the build times for core-image-sato during 1.5. I needed cut and paste data from the wiki and then clean it up in order to generate the chart. Before I continue I would like access to cleaner data. I hope the CSV files before they are transcribed to the wiki will be easier to work with. Can I please have copies of those? 36552 2 sstncr 2013-09-11 18:35:14 +0000 Hey Michael, The csv files generated by the build script look like this: thunderbird,master:2ab1bf27f79beadc45e86a26c85fc0b95ae3496f,1.5_M4.rc3-52-g2ab1bf2,1:09:32,12:27.33,1:10:23,5:25.02,0:32.23,0:17.07,0:01.63,26193184,5008012 thunderbird,master:2ab1bf27f79beadc45e86a26c85fc0b95ae3496f,1.5_M4.rc3-52-g2ab1bf2,1:09:06,12:15.77,1:09:55,5:23.13,0:32.62,0:17.01,0:01.64,26192144,5008020 thunderbird,master:0ca5d1fb38157564a2b9452ade32391d18a41b09,1.5_M4.rc3-57-g0ca5d1f,1:08:28,12:51.98,1:10:37,6:53.14,0:33.58,0:17.06,0:01.66,26187432,5006528 thunderbird,master:0ca5d1fb38157564a2b9452ade32391d18a41b09,1.5_M4.rc3-57-g0ca5d1f,1:09:15,12:52.84,1:09:39,5:12.10,0:32.13,0:17.13,0:01.63,26186328,5006488 thunderbird,master:a5973acca3224d64dd80a626febe5d84bd2e3bc4,1.5_M4.rc3-79-ga5973ac,1:08:35,11:58.80,1:15:06,5:24.62,0:32.08,0:16.98,0:01.62,26187900,5006480 thunderbird,master:6b4f7e4b26c51f0c1517c0a697031e7bc488b76f,1.5_M4.rc3-89-g6b4f7e4,1:08:49,12:21.03,1:10:29,5:17.03,0:32.08,0:17.03,0:01.63,26187480,5006604 (each build on a line, can be multiple builds on the same commit) I actually paste the csv into the wiki, see the source: https://wiki.yoctoproject.org/wiki/index.php?title=Performance_Test&action=edit And I just use <tab> </tab> to create a table with a header: <tab class=wikitable sep=comma head=top> hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work) thunderbird,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,dylan-9.0.1.rc1,1:05:52,11:04.48,1:05:23,5:16.83,0:30.80,0:17.16,0:02.01,28G,4.7G thunderbird,dylan:e4a0a4c5e419154a34710a1b6c28d4180c6304c3,dylan-9.0.1.rc1-6-ge4a0a4c,1:05:45,11:17.05,1:05:22,5:16.18,0:30.81,0:17.55,0:02.01,28G,4.7G </tab> 36615 3 sstncr 2013-09-12 18:02:17 +0000 So the raw data is in the wiki source: https://wiki.yoctoproject.org/wiki/index.php?title=Performance_Test&action=edit Does that help you? 36819 4 mhalstead 2013-09-17 17:36:17 +0000 I can complete the initial work using the data in the wiki. However this will not work long term. Moving data from a script into a wiki and then from that wiki to another script is unnecessarily complicated. Perhaps you could add a line to your script to push your data to a git repository which would be better for moving data between scripts. Would that work for you? 37024 5 sstncr 2013-09-20 14:22:13 +0000 Sure, I'll push it somewhere on poky-contrib and get back to you. 37877 6 alexandru.c.georgescu 2013-10-14 10:41:50 +0000 Hi Michael, Based on a discussion with RP the idea is that the charts to be displayed with error bars like this example: http://www.dplot.com/examples/error-bars.gif This way we will have an average but also the variance based on the different values obtained by running on the same commit. 45221 7 jefro 2014-08-22 20:02:31 +0000 No web content bugs should have a quarterly milestone. 58729 8 alexandru.c.georgescu 2016-02-08 13:52:52 +0000 The current implementation can be seen at: https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html 68085 9 markus.lehtonen 2016-11-08 13:17:51 +0000 I will prototype graphing of build perf test results on the CustomReports service: https://github.com/fullvlad/CustomReports 68095 10 benjamin.esquivel 2016-11-08 15:50:14 +0000 If you setup a staging instance of the CustomReports server please let me know when it can be seen. I'd like to watch your work there. 70332 11 markus.lehtonen 2017-02-01 15:21:33 +0000 Moving to milestone to 2.4 along with bug 6319 72287 12 markus.lehtonen 2017-04-11 12:31:18 +0000 Similarly to bug 6319, I'm setting the milestone now to 2.4 M1 but would be inclined to close the bug as fixed. Oe-core now has a tool (oe-build-perf-report) for generating graphical test reports, and, such reports are also being automatically generated and sent by the test infrastructure. I will split out the features that are not yet implemented into separate bug(s) before closing this one. 73444 13 markus.lehtonen 2017-05-23 07:51:47 +0000 Build performance test reports are now being automatically generated and uploaded to wiki.yoctoproject.org for each test run. Links to the latest test reports are found here: https://wiki.yoctoproject.org/wiki/Performance_Test#Daily_Test_Results Closing this bug. Missing features are tracked in separate bugs. 1442 2013-08-26 16:04:30 +0000 2013-08-26 16:04:30 +0000 Sample CSV file. sample.csv text/csv 970 alexandru.c.georgescu aG9zdG5hbWUsYnJhbmNoOmNvbW1pdCxnaXQgZGVzY3JpYmUsYml0YmFrZSBjb3JlLWltYWdlLXNh dG8sYml0YmFrZSB2aXJ0dWFsL2tlcm5lbCxiaXRiYWtlIGNvcmUtaW1hZ2Utc2F0byAocm1fd29y ayksYml0YmFrZSBjb3JlLWltYWdlLXNhdG8gLWMgcm9vdGZzLGJpdGJha2UgLXAgKHJtIC1yZiBj YWNoZS8gdG1wL2NhY2hlLyksIGJpdGJha2UgLXAgKHJtIC1yZiB0bXAvY2FjaGUvKSxiaXRiYWtl IC1wLHNpemUgb2YgdG1wIGRpciwgc2l6ZSBvZiB0bXAgZGlyIChybV93b3JrKQp1YnVudHUxMixk eWxhbjo3M2YxMDNiZjliMmNkZjk4NTQ2NGRjNTNiZjRmMWNmZDcxZDQ1MzFmLGR5bGFuLTkuMC4x LnJjMSwxOjIwOjE0LDExOjM4LjY5LDE6MTI6MDIsNToxOS42MSwwOjI3LjUxLDA6MTQuNzgsMDow Mi4wNiwyOEcsNC42Rwp1YnVudHUxMixkeWxhbjpkNzM0YWI0OTFhMzAwNzhkNDNkZWU1NDQwYzAz YWNjZTJkMjUxNDI1LHlvY3RvLTEuNC4xLTEyNS1nZDczNGFiNCwxOjE1OjM0LDEyOjUxLjQ0LDE6 MDk6NTcsNDo1NS4yMSwwOjI3LjMxLDA6MTUuMjgsMDowMi4wMSwyNTQyNzk1Miw0NzgzNjMyCnVi dW50dTEyLG1hc3RlcjphNjJhZWQ0MWYyZDhmODc0ZjdhZTI0ZDBlNWJlNWRiYzY2ZWEyMTk5LDEu NV9NMS5yYzEsMToyMDoxMCwxMjoyOC42NiwxOjEyOjI0LDU6MDAuNzgsMDoyNy45OCwwOjE1LjY0 LDA6MDIuMjAsMjVHLDQuNkcKdWJ1bnR1MTIsbWFzdGVyOmVhYTVkZjM0YWY0MmI2YTM3ZjY1MDY4 NDdkMGYzZWY2YmEwZDI5OGEsMS41X00yLnJjMSwxOjIwOjM5LDEzOjA2LjIwLDE6MTU6MjEsNDo0 NS4zMSwwOjI4LjM4LDA6MTcuMTYsMDowMi4wOSwyNUcsNC43Rwp1YnVudHUxMixtYXN0ZXI6OWRl MGFkNDc0NjJjMTNhYzRhMmQ3M2UxMmU5MmJlNmMxYjRlMDQxNSwxLjVfTTMucmMxLDE6MjA6MzQs MTI6NTcuMTIsMToxNjoxNCw0OjQ5LjYxLDA6MjguOTIsMDoxNi4xNSwwOjAyLjExLDI1Ryw0LjdH Cg==