Bug 9377

Could you elaborate what would be expected to be implemented by me? A set of new buildsteps, I suppose, but something else?

Please coordinate with Anibal for seeing his work on AB steps for testing and coordinate with Humberto for the AB Workers that will be used for performance measurements.

Assigning to Leo so he can redistribute it among the GDC team.

I finished a simple implementation, i need to test it before to send to the ML,

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?h=contrib/alimon/devel&id=bf0d3082a492d0ce110417e1ac321e477a79731d

Monse will take lead on this with Anibal's guidance.

Any updates for the patchset? I previously commented to Aníbal that it would be good to not use the build-perf-test-wrapper.sh script but oe-build-perf-test directly. The reason being that the wrapper script is quite stupid with many hardcoded bits. It would be more maintainable just to write out the steps performed by the wrapper in the autobuilder job

(In reply to comment #6)
> Any updates for the patchset? I previously commented to Aníbal that it would
> be good to not use the build-perf-test-wrapper.sh script but
> oe-build-perf-test directly. The reason being that the wrapper script is
> quite stupid with many hardcoded bits. It would be more maintainable just to
> write out the steps performed by the wrapper in the autobuilder job

Yes, I'm in sync with Monse and she has taken into account to don't use the wrapper.

Any status Monse?

(In reply to comment #7)
> (In reply to comment #6)
> > Any updates for the patchset? I previously commented to Aníbal that it would
> > be good to not use the build-perf-test-wrapper.sh script but
> > oe-build-perf-test directly. The reason being that the wrapper script is
> > quite stupid with many hardcoded bits. It would be more maintainable just to
> > write out the steps performed by the wrapper in the autobuilder job
> 
> Yes, I'm in sync with Monse and she has taken into account to don't use the
> wrapper.
> 
> Any status Monse?

Yes, I am working on that patchset, and we are not use the wrapper script.

In review at,

https://lists.yoctoproject.org/pipermail/yocto/2017-January/033937.html

Buildset is integrated into Autobuilder master,

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?id=3406824a7cdfd256db1f9fa55b3700df70d77ce7

We will work on deployment on GDC AB's.

Reopening this as this was never actually deployed and it is broken in its current state as the command line interface of oe-build-perf-test has changed. Git operations should be handled by the oe-git-archive script. Also, we should xml report format.

Moving to M3 since Markus is on leave.

(In reply to comment #11)
> Reopening this as this was never actually deployed and it is broken in its
> current state as the command line interface of oe-build-perf-test has
> changed. Git operations should be handled by the oe-git-archive script.
> Also, we should xml report format.

Looking at the existing oe-build-perf-test call:

""" oe-build-perf-test --out-dir "%s" --globalres-file "%s" --lock-file "%s" --commit-results "%s" --commit-results-branch "{tester_host}/{git_branch}/%s" --commit-results-tag "{tester_host}/{git_branch}/%s/{git_commit_count}-g{git_commit}/{tag_num}" """ % (results_dir, globalres_log, lock_file, git_dir, machine, machine)

I think the new oe-build-perf-test invocation should look like:

"oe-build-perf-test --outdir %s --globalres-file %s --lock-file %s -x" % (results_dir, globalres_log, lock_file)

Could you help us get the correct invocation semantics for oe-git-archive?

(In reply to comment #13)
> 
> Looking at the existing oe-build-perf-test call:
> 
> """ oe-build-perf-test --out-dir "%s" --globalres-file "%s" --lock-file "%s"
> --commit-results "%s" --commit-results-branch
> "{tester_host}/{git_branch}/%s" --commit-results-tag
> "{tester_host}/{git_branch}/%s/{git_commit_count}-g{git_commit}/{tag_num}"
> """ % (results_dir, globalres_log, lock_file, git_dir, machine, machine)
> 
> I think the new oe-build-perf-test invocation should look like:
> 
> "oe-build-perf-test --outdir %s --globalres-file %s --lock-file %s -x" %
> (results_dir, globalres_log, lock_file)

Yes, this looks correct.

 
> Could you help us get the correct invocation semantics for oe-git-archive?

Looking at scripts/contrib/build-perf-test-wrapper.sh, it would be something like (in Popen-compatible list format):
[ "oe-git-archive", 
  "--git-dir", results_repo,
  "--branch-name", "{hostname}/{branch}/{machine}",
  "--tag-name", "{hostname}/{branch}/{machine}/{commit_count}-g{commit}/{tag_number}",
  "--exclude", "buildstats.json",
  "--notes", "buildstats/{branch_name}", os.path.join(results_dir, "buildstats.json"),
  "--push", git_remote,
  results_dir ]

Where:
- 'results_dir' is the output dir used for oe-build-perf-test
- 'results_repo' is the local git repo to commit the results in
- 'git_remote' is the remote where to push results

The '--exclude' option makes buildstats not to be committed into git. But, the '--notes' option (with two arguments, notes ref and the file to be committed) makes the buildstats to be saved in git-notes, instead.

In addition to running the test and archiving the results, the reporting functionality (email reporting and uploading html report to a remote location) of build-perf-test-wrapper.sh should be implemented in autobuilder.

(In reply to comment #15)
> In addition to running the test and archiving the results, the reporting
> functionality (email reporting and uploading html report to a remote
> location) of build-perf-test-wrapper.sh should be implemented in autobuilder.

Is there any reason we shouldn't just use build-perf-test-wrapper.sh directly?

(In reply to comment #16)
> (In reply to comment #15)
> > In addition to running the test and archiving the results, the reporting
> > functionality (email reporting and uploading html report to a remote
> > location) of build-perf-test-wrapper.sh should be implemented in autobuilder.
> 
> Is there any reason we shouldn't just use build-perf-test-wrapper.sh
> directly?

I just think it's hard to maintain. And, against the autobuilder idea of separating the build into separate simpler buildsteps. The wrapper script contains all steps in one easily breaking shell script with a lot of hardcoded assumptions. My assumption was that using autobuilder would simplify things, e.g. being able to re-use readily available bits for fetching, initializing build dir and config etc.

Of course you can just call the wrapper script, though. Maybe that would be a logical initial implementation

Bumping this to 2.5. Running the tests and generating the reports is easy thanks to the scripts Markus has developed in OE-Core.

Translating the reports for publishing requires some more thought and work, I'm reticent to require phantomjs (an abandoned project) on all workers that might send mail, but equally I'm in no hurry to implement an alternative on the autobuilder.

It might make sense to port the phantomjs usage of OE-Core's scripts/contrib/oe-build-perf-report-email.py to slimerjs[1]/headless firefox[2] or something using Chrome's Puppeteer[3][4]

1. https://slimerjs.org/
2. https://developer.mozilla.org/en-US/Firefox/Headless_mode
3. https://github.com/GoogleChrome/puppeteer
4. https://medium.com/@kensoh/chromeless-chrominator-chromy-navalia-lambdium-ghostjs-autogcd-ef34bcd26907

Using phantomjs is probably a bit unfortunate choice. I just chose it semi-randomly to get a prototype of the whole reporting system in shape and running.

But, it's quite thinly coupled with build-perf-report-email.py and even the phantomjs script is fairly short and simple (it basically looks at certain messages in the javascript console and grabs chart PNGs as they are being rendered). Thus, it should be relatively easy to replace it with something else.

Slimerjs would probably be the easiest switch. Chrome would be another good option with a large community

We now run build performance tests in the autobuilder