9377 2016-04-01 20:52:48 +0000 Enable automated build performance testing in Autobuilder 2019-02-28 15:15:02 +0000 1 1 10 QA/Testing Build Testing general unspecified x86 Multiple RESOLVED FIXED Medium+ enhancement Future 13014 9769 1 jose.perez.carranza richard.purdie anibal.limon benjamin.esquivel elizabeth.flanagan joshuagloe leonardo.sandoval.gonzalez markus.lehtonen richard.purdie ross.burton sgw stephano --- --- No (bug/feature does not impact docs) oldest_to_newest 60709 0 jose.perez.carranza 2016-04-01 20:52:48 +0000 Currently the results for build performance tests are sent by mail with a raw data that is not easy to manage, in order to have better control of the results we need to store them in a standard and easy to mange format, the first approach to do this is to enable the capability on Autobuilder to run performance test jobs in a specific list of machines. 62293 1 markus.lehtonen 2016-05-18 13:44:23 +0000 Could you elaborate what would be expected to be implemented by me? A set of new buildsteps, I suppose, but something else? 62489 2 benjamin.esquivel 2016-05-26 20:23:24 +0000 Please coordinate with Anibal for seeing his work on AB steps for testing and coordinate with Humberto for the AB Workers that will be used for performance measurements. 67823 3 benjamin.esquivel 2016-11-01 16:50:01 +0000 Assigning to Leo so he can redistribute it among the GDC team. 68052 4 anibal.limon 2016-11-07 22:54:48 +0000 I finished a simple implementation, i need to test it before to send to the ML, http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?h=contrib/alimon/devel&id=bf0d3082a492d0ce110417e1ac321e477a79731d 68418 5 leonardo.sandoval.gonzalez 2016-11-17 18:02:02 +0000 Monse will take lead on this with Anibal's guidance. 68434 6 markus.lehtonen 2016-11-18 07:13:03 +0000 Any updates for the patchset? I previously commented to Aníbal that it would be good to not use the build-perf-test-wrapper.sh script but oe-build-perf-test directly. The reason being that the wrapper script is quite stupid with many hardcoded bits. It would be more maintainable just to write out the steps performed by the wrapper in the autobuilder job 68441 7 anibal.limon 2016-11-18 14:47:50 +0000 (In reply to comment #6) > Any updates for the patchset? I previously commented to Aníbal that it would > be good to not use the build-perf-test-wrapper.sh script but > oe-build-perf-test directly. The reason being that the wrapper script is > quite stupid with many hardcoded bits. It would be more maintainable just to > write out the steps performed by the wrapper in the autobuilder job Yes, I'm in sync with Monse and she has taken into account to don't use the wrapper. Any status Monse? 68442 8 monserratx.sedeno.bustos 2016-11-18 16:31:42 +0000 (In reply to comment #7) > (In reply to comment #6) > > Any updates for the patchset? I previously commented to Aníbal that it would > > be good to not use the build-perf-test-wrapper.sh script but > > oe-build-perf-test directly. The reason being that the wrapper script is > > quite stupid with many hardcoded bits. It would be more maintainable just to > > write out the steps performed by the wrapper in the autobuilder job > > Yes, I'm in sync with Monse and she has taken into account to don't use the > wrapper. > > Any status Monse? Yes, I am working on that patchset, and we are not use the wrapper script. 69827 9 anibal.limon 2017-01-16 16:24:53 +0000 In review at, https://lists.yoctoproject.org/pipermail/yocto/2017-January/033937.html 70058 10 anibal.limon 2017-01-23 22:34:50 +0000 Buildset is integrated into Autobuilder master, http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=3406824a7cdfd256db1f9fa55b3700df70d77ce7 We will work on deployment on GDC AB's. 73143 11 markus.lehtonen 2017-05-09 14:19:40 +0000 Reopening this as this was never actually deployed and it is broken in its current state as the command line interface of oe-build-perf-test has changed. Git operations should be handled by the oe-git-archive script. Also, we should xml report format. 75151 12 stephano 2017-07-19 19:37:01 +0000 Moving to M3 since Markus is on leave. 75719 13 joshuagloe 2017-08-08 12:59:48 +0000 (In reply to comment #11) > Reopening this as this was never actually deployed and it is broken in its > current state as the command line interface of oe-build-perf-test has > changed. Git operations should be handled by the oe-git-archive script. > Also, we should xml report format. Looking at the existing oe-build-perf-test call: """ oe-build-perf-test --out-dir "%s" --globalres-file "%s" --lock-file "%s" --commit-results "%s" --commit-results-branch "{tester_host}/{git_branch}/%s" --commit-results-tag "{tester_host}/{git_branch}/%s/{git_commit_count}-g{git_commit}/{tag_num}" """ % (results_dir, globalres_log, lock_file, git_dir, machine, machine) I think the new oe-build-perf-test invocation should look like: "oe-build-perf-test --outdir %s --globalres-file %s --lock-file %s -x" % (results_dir, globalres_log, lock_file) Could you help us get the correct invocation semantics for oe-git-archive? 75732 14 markus.lehtonen 2017-08-08 15:41:27 +0000 (In reply to comment #13) > > Looking at the existing oe-build-perf-test call: > > """ oe-build-perf-test --out-dir "%s" --globalres-file "%s" --lock-file "%s" > --commit-results "%s" --commit-results-branch > "{tester_host}/{git_branch}/%s" --commit-results-tag > "{tester_host}/{git_branch}/%s/{git_commit_count}-g{git_commit}/{tag_num}" > """ % (results_dir, globalres_log, lock_file, git_dir, machine, machine) > > I think the new oe-build-perf-test invocation should look like: > > "oe-build-perf-test --outdir %s --globalres-file %s --lock-file %s -x" % > (results_dir, globalres_log, lock_file) Yes, this looks correct. > Could you help us get the correct invocation semantics for oe-git-archive? Looking at scripts/contrib/build-perf-test-wrapper.sh, it would be something like (in Popen-compatible list format): [ "oe-git-archive", "--git-dir", results_repo, "--branch-name", "{hostname}/{branch}/{machine}", "--tag-name", "{hostname}/{branch}/{machine}/{commit_count}-g{commit}/{tag_number}", "--exclude", "buildstats.json", "--notes", "buildstats/{branch_name}", os.path.join(results_dir, "buildstats.json"), "--push", git_remote, results_dir ] Where: - 'results_dir' is the output dir used for oe-build-perf-test - 'results_repo' is the local git repo to commit the results in - 'git_remote' is the remote where to push results The '--exclude' option makes buildstats not to be committed into git. But, the '--notes' option (with two arguments, notes ref and the file to be committed) makes the buildstats to be saved in git-notes, instead. 75757 15 markus.lehtonen 2017-08-09 04:40:45 +0000 In addition to running the test and archiving the results, the reporting functionality (email reporting and uploading html report to a remote location) of build-perf-test-wrapper.sh should be implemented in autobuilder. 76890 16 joshuagloe 2017-09-15 13:37:36 +0000 (In reply to comment #15) > In addition to running the test and archiving the results, the reporting > functionality (email reporting and uploading html report to a remote > location) of build-perf-test-wrapper.sh should be implemented in autobuilder. Is there any reason we shouldn't just use build-perf-test-wrapper.sh directly? 76891 17 markus.lehtonen 2017-09-15 14:03:57 +0000 (In reply to comment #16) > (In reply to comment #15) > > In addition to running the test and archiving the results, the reporting > > functionality (email reporting and uploading html report to a remote > > location) of build-perf-test-wrapper.sh should be implemented in autobuilder. > > Is there any reason we shouldn't just use build-perf-test-wrapper.sh > directly? I just think it's hard to maintain. And, against the autobuilder idea of separating the build into separate simpler buildsteps. The wrapper script contains all steps in one easily breaking shell script with a lot of hardcoded assumptions. My assumption was that using autobuilder would simplify things, e.g. being able to re-use readily available bits for fetching, initializing build dir and config etc. Of course you can just call the wrapper script, though. Maybe that would be a logical initial implementation 77194 18 joshuagloe 2017-09-28 11:21:15 +0000 Bumping this to 2.5. Running the tests and generating the reports is easy thanks to the scripts Markus has developed in OE-Core. Translating the reports for publishing requires some more thought and work, I'm reticent to require phantomjs (an abandoned project) on all workers that might send mail, but equally I'm in no hurry to implement an alternative on the autobuilder. It might make sense to port the phantomjs usage of OE-Core's scripts/contrib/oe-build-perf-report-email.py to slimerjs[1]/headless firefox[2] or something using Chrome's Puppeteer[3][4] 1. https://slimerjs.org/ 2. https://developer.mozilla.org/en-US/Firefox/Headless_mode 3. https://github.com/GoogleChrome/puppeteer 4. https://medium.com/@kensoh/chromeless-chrominator-chromy-navalia-lambdium-ghostjs-autogcd-ef34bcd26907 77224 19 markus.lehtonen 2017-09-29 07:41:08 +0000 Using phantomjs is probably a bit unfortunate choice. I just chose it semi-randomly to get a prototype of the whole reporting system in shape and running. But, it's quite thinly coupled with build-perf-report-email.py and even the phantomjs script is fairly short and simple (it basically looks at certain messages in the javascript console and grabs chart PNGs as they are being rendered). Thus, it should be relatively easy to replace it with something else. Slimerjs would probably be the easiest switch. Chrome would be another good option with a large community 83153 20 richard.purdie 2019-02-28 15:15:02 +0000 We now run build performance tests in the autobuilder