We should agree on a test report format that would be most suitable for the upcoming test report tool. We also need to consider that all the data will be stored in Git. The filesystem layout of the test result data currently look like this: results-YYYYMMDDhhmmss/ ├── build │ └── conf │ ├── local.conf │ └── ... ├── oe-build-perf-test.log ├── results.json ├── test1 │ ├── buildstats.bitbake-quilt-native.json │ └── commands.log ├── test12 │ └── ... └── ... Of these: - build/conf contains the bitbake configuration file(s) used in the test run. - oe-build-perf-test.log is a log of the oe-build-perf-test script itself - results.json contains the actual test report - test specific directories contain two types of items: * commands.log is contains output of all commands run in the test case * possible buildstats in json format Some questions about the report format: - is the file layout practical? - is JSON good for us? - details of the JSON report * what data to put there * how data is organized inside the file * should we have one consolidated report (as now) or per-test json files
Currently, the results.json looks like this: { "git_branch": "master", "git_commit": "3c9b207526455a3257fb67eff8eef253be2e96f0", "git_commit_count": "33881", "product": "oe-core", "tester_host": "mqz-suse64", "start_time": 1478163073.502729, "elapsed_time": 65.318164, "tests": { "test1": { "name": "test1", "description": "Measure wall clock of bitbake core-image-sato and size of tmp dir", "start_time": 1478170273.502802, "elapsed_time": 19.256773, "status": "SUCCESS" "cmd_log_file": "test1/commands.log", "measurements": [ { "name": "build", "legend": "bitbake core-image-sato", "type": "sysres", "values": { "buildstats_file": "test1/buildstats.bitbake-quilt-native.json", "elapsed_time": 3.502545, "iostat": { "cancelled_write_bytes": 24576, "rchar": 28896074, "read_bytes": 45056, "syscr": 71565, "syscw": 1699, "wchar": 4435543, "write_bytes": 380928 }, "rusage": { "ru_inblock": 88, "ru_majflt": 0, "ru_maxrss": 106852, "ru_minflt": 90725, "ru_nivcsw": 1515, "ru_nvcsw": 10545, "ru_oublock": 744, "ru_stime": 0.46799999999999997, "ru_utime": 3.868 }, "start_time": 1478170276.250525 } }, { "legend": "tmpdir", "name": "tmpdir", "type": "diskusage", "values": { "size": 16256080 } } ], }, "test12": { "name": "test12", "description": "Measure bitbake virtual/kernel", "cmd_log_file": "test12/commands.log", "start_time": 1478170310.342975, "elapsed_time": 5.523124, "status": "FAILURE" "err_output": "Traceback (most recent call last): ..." "err_type": "AssertionError", "message": "", "measurements": [], }, ... } }
(In reply to comment #0) ... > > Some questions about the report format: > - is the file layout practical? It looks like a good start. Remains to be seen whether the TRT imposes some changes here. > - is JSON good for us? I currently don't see a reason not to use it. > - details of the JSON report > * what data to put there > * how data is organized inside the file writing a consumer parser helps you know if you need a different structure or this one is okay. > * should we have one consolidated report (as now) or per-test json files if the consolidated json is the same as the sum of the individual reports then we could live without one of them. Granularity matters so I would choose the per-test to stay.
(In reply to comment #2) > (In reply to comment #0) > ... > > > > Some questions about the report format: > > - is the file layout practical? > It looks like a good start. Remains to be seen whether the TRT imposes some > changes here. > > - is JSON good for us? > I currently don't see a reason not to use it. OK, good. > > - details of the JSON report > > * what data to put there > > * how data is organized inside the file > writing a consumer parser helps you know if you need a different structure > or this one is okay. Hmm, what do you mean with consumer parser? A parser for TRT or smth else? What I was thinking, for example, should we have the test descriptions in the report or try to fetch them from somewhere else. Probably it's good to have them in the report(?) Another thing is what information about the host system should be incorporated. Now there's only hostname. We probably be good to include e.g. name and version from /etc/os-release. > > * should we have one consolidated report (as now) or per-test json files > if the consolidated json is the same as the sum of the individual reports > then we could live without one of them. Granularity matters so I would > choose the per-test to stay. I meant to say per test case json files. Now we have one monolithic json file for one test run. But I guess it doesn't matter that much.
I think this bug should be assigned to Markus and me as QA contact, doing the change now.
I had some talk with Benjamin about xml format. I have an rfc patchset for generating xml reports: http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/xml
The XML format produced by the patchset is JUnit-like but extended with "non-standard" elements. Most notably: - git revision and product information as attributes of <testsuites> element - test description attribute in <testcase> - <sysres> and <diskusage> elements under <testcase> <?xml version="1.0" encoding="utf-8"?> <testsuites git_branch="feature/buildperf/xml" git_commit="2d012870bfc43c4aa55cd96b55c1c36f9e6ba218" git_commit_count="34073" product="oe-core"> <testsuite errors="1" failures="1" hostname="marquiz" name="oeqa.buildperf" skipped="1" tests="6" time="22.347736" timestamp="2016-11-14T11:18:03.419477"> <testcase classname="oeqa.buildperf.test_basic.Test1P1" description="Measure wall clock of bitbake core-image-sato and size of tmp dir" name="test1" time="4.575692" timestamp="2016-11-14T13:18:03.419602"> <sysres legend="bitbake quilt-native" name="build"> <time>2.269468</time> <buildstats_file>test1/buildstats.bitbake-quilt-native.json</buildstats_file> <iostat cancelled_write_bytes="24576" rchar="33302047" read_bytes="0" syscr="53293" syscw="1717" wchar="4318553" write_bytes="327680"/> <rusage ru_inblock="0" ru_majflt="0" ru_maxrss="112232" ru_minflt="101886" ru_nivcsw="84" ru_nvcsw="9542" ru_oublock="640" ru_stime="0.16799999999999998" ru_utime="2.488"/> </sysres> <diskusage legend="tmpdir" name="tmpdir"> <size>32044</size> </diskusage> </testcase> ···· <testcase classname="oeqa.buildperf.test_basic.Test1P2" description="Measure bitbake virtual/kernel" name="test12" time="4.430943" timestamp="2016-11-14T13:18:07.995385"> <sysres legend="bitbake quilt-native" name="build"> <time>2.219579</time> <iostat cancelled_write_bytes="24576" rchar="33302771" read_bytes="0" syscr="52540" syscw="1717" wchar="4319267" write_bytes="327680"/> <rusage ru_inblock="0" ru_majflt="0" ru_maxrss="112060" ru_minflt="101833" ru_nivcsw="80" ru_nvcsw="9478" ru_oublock="640" ru_stime="0.208" ru_utime="2.408"/> </sysres> </testcase> ···· <testcase classname="oeqa.buildperf.test_basic.Test1P3" description="" name="test13" time="2.278677" timestamp="2016-11-14T13:18:12.426471"> <failure message="" type="AssertionError">Traceback (most recent call last): File "/home/marquiz/yocto/openembedded-core/meta/lib/oeqa/buildperf/test_basic.py", line 54, in test13 assert False AssertionError </failure> </testcase> ... </testsuite> </testsuites>
Assigning to Benjamin for comments from QA team.
After review the structure of the proposal looks goo to me
Markus, From the last comment, I don't believe this is still NEEDINFO. Please put it back in NEEDINFO if required.
I put the xml proposal here: https://wiki.yoctoproject.org/wiki/QA/Automated_Test_Report_Template#Execution_Information_Template I also removed the product and git revision information from the example there. These should be part of the "metadata" xml format that is also outlined in the same wiki page.
Submitted for review: https://patchwork.openembedded.org/series/4840/
Merged in oe-core: http://git.openembedded.org/openembedded-core/commit/?id=51ae18c64eee074478157a484f71a53faadca80b ... http://git.openembedded.org/openembedded-core/commit/?id=81065092f38c9631dcf5917d70a25809a21de5f4