We would need to make build performance test results more easily available. Storing all the test data in Git is a good solution for now: - the data is easily available for everybody - we can utilize existing services and protocols (we do not need yet another database somewhere, for example) We would need: 1. an external repository for the data, e.g. git.yoctoproject.org/poky-build-perf-data 2. decide on branching/tagging 3. start pushing results there The oe-build-perf-test script currently has the functionality to store test data in a local git repository (--commit-results, --commit-results-branch, --commit-results-tag). It is not capable of pushing the results to a remote repo. That functionality could be added to build-perf-test-wrapper.sh, for example. Regarding branching and tagging, the defaults used by build-perf-test-wrapper.sh are Branch name: <HOSTNAME>/<BRANCH>/<MACHINE> Tag name: <HOSTNAME}/<GIT_BRANCH>/<MACHINE>/<COMMIT_COUNT>-g<GIT_COMMIT>/<TAG_NUM> Where: HOSTNAME is the hostname of the tester machine BRANCH is the target branch we're testing, e.g. master or morty MACHINE is the target machine COMMIT_COUNT is a sequence number of the commit that was tested (i.e. number of commits since the initial commit) GIT_COMMIT is the SHA1 checksum of the commit that was tested TAG_NUM is an increasing number to separate tests on the same commit The idea behind tagging each result is to provide an easier way to filtering test data, e.g. list tested revisions in git history order or list all tests for certain revision. This would also make it more feasible to run tests in non-chronological order, e.g. test a revision C between A and B, after both A and B has been tested, and add test results of C into the repo.
Actually, before starting to push new results, we should push all existing old results into the Git repository.
adding José to the bug. José, I presume you have the old results from the performance runs, this in order to push them to the git repo of the results like Markus mentions.
I have a script in the works that converts old archived results in the new format and stores them in Git. I'll share it with you soon after I've done some testing on it.
Created attachment 3522 [details] old performance measurements for Ubuntu
Created attachment 3523 [details] old performance measurements for Fedora
Writing a universal script was a bit trickier than I first thought. For example, the output format of different bits and pieces has changed over time and there are various kinds of invalid/failed test runs among the result archives. However, I now have a script here that seems to do quite decent job: http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/git-import Jose: the "globalres" log files you attached are not enough. The archived tarballs need to be used as they contain way more data than just the "top-level" results. Please try out my script and see how it works for you. The script currently converts test results into similar JSON format that oe-build-perf-test produces. The format is described in bug10590. When bug10590 gets solved we should be able to go into production (at least in local Git level): [0. Adjust oe-build-perf-test to produce the agreed output format] 1. I'll adjust the import script to produce correct output format 2. Stop testers 3. Import all old results into a Git repo 4. Enable testers with automatic git storage enabled
Do you have any comments on the branch and tag naming scheme I suggested?
Oh, I forgot to give an example how to use the git-import script: $ scripts/contrib/build-perf-git-import.py -c -P ~/poky -g ~/poky-perf-archive.git ~/poky-perf/archives/*tag.gz
Assigning to Benjamin for comments from QA team. Any comments on branching / tagging scheme? Any comments or feedback on the git import script I wrote?
I will be out for some days and I've instructed Mariano to follow up on this. I've also updated him with our conversations so it should be relatively easy to make progress on this task.
Was talking with Marion about the format of the branch an the tags and we agreed that the current proposal is the adequate for our needs.
(In reply to comment #11) > Was talking with Marion about the format of the branch an the tags and we > agreed that the current proposal is the adequate for our needs. I mean Mariano on the previous comment !!
Assigning to QA team (Mariano), please re-assign if somebody else would be better fit. Bug 10590 is soon (hopefully) fixed and after that the steps would be to enable Git storage on the QA side. That would be using the '--commit-results' option of oe-build-perf-test script or -C option of the wrapper script. However, before enabling Git storage, the old archived results should be imported into a Git repository. This can be done with my build-perf-git-import script below.
Assigning to Jose, can you check on Markus comments?
Richard The git storage that needs to be under GDC infrastructure or should be under the yocto project infrastructure - git.yoctoproject.org - ? depending on this decision this bug should be reassigned to the proper owner.
I think the results should be under git.yoctoproject.org That is, the tester committing into local git and then pushing to git.yoctoproject.org (as a separate build step).
The repositories should definitely live on git.yp.o Afaict the next step is to set up the repositories on git.yp.o and perform the initial import?
The repos should be on git.yoctoproject.org, reassigning to Joshua to work with Michael/Markus/whoever else is needed to resolve this.
An attempt to summarise: DONE: 1. repository format has been agreed on 2. a script exists to convert old data to the new format and commit it to a git repository TODO: 1. convert old data to new format — Jose to own? 2. create public repository to house build perf data — Joshua to own? 3. implement appropriate tooling to have the new data pushed automatically — Markus to own? Related to 2: * who should be able to push to that repo? just a user on the AB infrastructure? * I'd like the repo names for QA data to be namespaced consistently, I'm thinking yp-qa-build-perf-data, yp-qa-selftest-data, etc. Any objections/comments?
(In reply to comment #19) > An attempt to summarise: > > DONE: > 1. repository format has been agreed on > 2. a script exists to convert old data to the new format and commit it to a > git repository > > TODO: > 1. convert old data to new format — Jose to own? I can coordinate this with Jose. > 2. create public repository to house build perf data — Joshua to own? > 3. implement appropriate tooling to have the new data pushed automatically — > Markus to own? I can write a helper script for committing pushing results. However, an autobuilder step for calling that script would be needed as well, Jose(?) > Related to 2: > * who should be able to push to that repo? just a user on the AB > infrastructure? I have resurrected some old build perf qa machines and would like to be able to push their results, too. They are not part of AB, at least not yet. > * I'd like the repo names for QA data to be namespaced consistently, I'm > thinking yp-qa-build-perf-data, yp-qa-selftest-data, etc. Any > objections/comments? I agree on consistent naming. My only comment is that other repos in git.yp.o seem to have poky-prefix. So maybe poky-qa-build-perf-data(?)
(In reply to comment #20) > (In reply to comment #19) > > 3. implement appropriate tooling to have the new data pushed automatically — > > Markus to own? > > I can write a helper script for committing pushing results. However, an > autobuilder step for calling that script would be needed as well, Jose(?) I can take care of the autobuilder integration. We might want a child bug for that. > > * I'd like the repo names for QA data to be namespaced consistently, I'm > > thinking yp-qa-build-perf-data, yp-qa-selftest-data, etc. Any > > objections/comments? > > I agree on consistent naming. My only comment is that other repos in > git.yp.o seem to have poky-prefix. So maybe poky-qa-build-perf-data(?) I'd like to stick with yp- as this is Yocto Project data and often not specific to the poky distro, if we were doing buildhistory again today we'd certainly call it yp-buildhistory.
(In reply to comment #21) > (In reply to comment #20) > > (In reply to comment #19) > > > 3. implement appropriate tooling to have the new data pushed automatically — > > > Markus to own? > > > > I can write a helper script for committing pushing results. However, an > > autobuilder step for calling that script would be needed as well, Jose(?) > > I can take care of the autobuilder integration. We might want a child bug > for that. I quickly wrote a helper script for committing data to git and pushing it upstream (cannibalizing some code from oe-build-perf test). Please take a look, try it out and comment, you can find the code here: http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-10582 It supports flexible (pattern-based) tag and branch naming as well as flexible commit and tag messages. It is also able to directly work with bare repos which should be less error-prone (no working copy to mess-up with). I'll send it for review after a bit more testing and integrating it to build perf test scripts. > > > * I'd like the repo names for QA data to be namespaced consistently, I'm > > > thinking yp-qa-build-perf-data, yp-qa-selftest-data, etc. Any > > > objections/comments? > > > > I agree on consistent naming. My only comment is that other repos in > > git.yp.o seem to have poky-prefix. So maybe poky-qa-build-perf-data(?) > > I'd like to stick with yp- as this is Yocto Project data and often not > specific to the poky distro, if we were doing buildhistory again today we'd > certainly call it yp-buildhistory. No objection here, that sounds reasonable. I was just looking at the old, obviously ill-named, repos :)
(In reply to comment #22) > (In reply to comment #21) > > (In reply to comment #20) > > > (In reply to comment #19) > > > > 3. implement appropriate tooling to have the new data pushed automatically — > > > > Markus to own? > > > > > > I can write a helper script for committing pushing results. However, an > > > autobuilder step for calling that script would be needed as well, Jose(?) > > > > I can take care of the autobuilder integration. We might want a child bug > > for that. > > I quickly wrote a helper script for committing data to git and pushing it > upstream (cannibalizing some code from oe-build-perf test). Please take a > look, try it out and comment, you can find the code here: > http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes- > 10582 > > It supports flexible (pattern-based) tag and branch naming as well as > flexible commit and tag messages. It is also able to directly work with bare > repos which should be less error-prone (no working copy to mess-up with). > > I'll send it for review after a bit more testing and integrating it to build > perf test scripts. Forgot to mention that the helper script needs to be run under an initialized build directory because it uses 'metadata' (machine, git revision info etc) for string formatting in branch and tag naming.
I sent my "git management" script for review: https://patchwork.openembedded.org/series/5208/
Looks like Markus' patches are in. We need a branch to push and a repository to push to. The latter I can take care of, any progress on the conversion of old data?
Markus/Jose — where are we with data conversion? Can we get these repositories published before the end of M4?
(In reply to comment #26) > Markus/Jose — where are we with data conversion? Can we get these > repositories published before the end of M4? Markus are you able to help on this conversion, I think you can stop the cron of the perf machines to do it.
Size of the Git repository became a concern. With thousands of test runs it took gigabytes of space. In order to mitigate this, I wrote one more patchset: https://patchwork.openembedded.org/series/5953/ Once this has been merged we should finally be ready and I can (re-)import the old result data to Git. My import script has already been adjusted accordingly.
Markus is driving all of the work here, so reassigning to him
All related patches have now been merged to oe-core. Deployment is still underway. I'm keeping this open until data is available at git://git.yoctoproject.org/yp-qa-build-perf-data.git
Deployment has been completed. We're just missing the public repo git://git.yoctoproject.org/yp-qa-build-perf-data.git Assigning to Joshua.
All code changes merged, perf data being pushed and the repository is now public: http://git.yoctoproject.org/clean/cgit.cgi/yp-qa-build-perf-data/