9954 2016-07-15 16:00:02 +0000 report tool: a communication mechanism for sending results 2017-12-06 17:03:36 +0000 1 1 10 QA/Testing Test Plans/Suite test-plans/suite 2.2 x86 Multiple RESOLVED FIXED https://bugzilla.yoctoproject.org/show_bug.cgi?id=9559 https://bugzilla.yoctoproject.org/show_bug.cgi?id=9682 High enhancement 2.3 M1 9682 10162 10639 1 benjamin.esquivel mariano.lopez benjamin.esquivel bluelightning corneliux.stoicescu joshuagloe leonardo.sandoval.gonzalez mariano.lopez markus.lehtonen michael.g.wood ross.burton sgw --- --- Don't know oldest_to_newest 64051 0 benjamin.esquivel 2016-07-15 16:00:02 +0000 For the test reporting tool to consume results, it must be able to receive them from a remote source. Once bug 9682 or similar is implemented, it means that the test results are in a standard format and it can be received at the test reporting server. The goal is to define a mechanism for passing results from a test client to a reporting server with the following requirements in mind: - Client authentication - report sessions - ability to resume - ability to update existing results - ability to abort/discard - reuse existing communication protocols (xml-rpc, json-rpc, etc) This feature can be demonstrated without depending on bug 9682 by using python's unittest xml or json text test runners with sample tests and using the following procedure: 1. at the client, with a results file in json/xml, issue a remote call to a known test server notifying the desire to establish a communication session 2. at the server, listen for client calls and grant them communication sessions if they pass the authentication challenge. 3. at the server, receive results from the session with the client. Store the results locally. 64878 1 joshuagloe 2016-08-04 15:42:09 +0000 I think we could make our lives easier in many ways if we have the testing client push data to a repository and the TRT consume data from that repository. This would help because: * We remove linkage between the client and the TRT, meaning we don't have to update them in sync, we don't have to ensure the TRT has amazing uptime, etc. * We keep the raw test data and can use it in alternative ways that we haven't thought of yet * We don't have to worry about security of an auth interface on the TRT * We don't have an API which must be protected against attacks I'd envisaged a system where: a) QA data are pushed to git (much like buildhistory data in poky-buildhistory[1]) b) Some mechanism such as a post-commit hook, DataKit[2], FedMsg[3], a simple timer/cron job etc let's the TRT know there's new data to process c) The TRT database is updated with the new data It's possible that the piece at step b) should process the raw data and submit it to the TRT, and therefore that we'd still need this communication mechanism, though that would still require an auth mechanism and DoS mitigations. I'm certain, however, that we don't want direct linkage between the testing client (i.e. AB workers) and the TRT, and also that we should preserve the raw data (buildhistory, selftest reports, performance data). 1. http://git.yoctoproject.org/cgit/cgit.cgi/poky-buildhistory/ 2. https://github.com/docker/datakit 3. http://www.fedmsg.com/en/latest/overview/ 65031 2 benjamin.esquivel 2016-08-10 18:25:59 +0000 I like the proposal, if the repository is treated as intermediate and no future data coherency is expected. Say, that would avoid problems like when results are erased, it would not be expected these results are erased from the repository as well. 65032 3 benjamin.esquivel 2016-08-10 18:27:12 +0000 re-assigning to Leo so he can find someone in the team to implement it. 65144 4 leonardo.sandoval.gonzalez 2016-08-12 19:27:43 +0000 If I understand correctly, this bug is about this data flow AB -> nightly-oe-selftest -> XML files > git add XML files -> git commit -> git push The interesting point here is to define the repository tree structure. Subsequent bugs derived from this one: * step b on comment https://bugzilla.yoctoproject.org/show_bug.cgi?id=9954#c1: This will push the XML into the TRT somehow. * The TRT itself if it does not exist. 65203 5 mariano.lopez 2016-08-15 20:09:54 +0000 Should this be implemented as another option of oe-selftest or should it push everytime it runs? 65241 6 joshuagloe 2016-08-16 20:04:45 +0000 We could have oe-selftest include an option to write output and commit them to a git repository, or we could rely on whatever's calling oe-selftest (i.e. the autobuilder) to collect the oe-selftest output and commit it to a repository. The closest analogue we currently have is buildhistory, which has logic to push to create a local git repo of the buildhistory and push it to a remote based on OE configuration variables. Would it make sense to do similar for oe-selftest? 65283 7 benjamin.esquivel 2016-08-17 23:15:42 +0000 (In reply to comment #6) > We could have oe-selftest include an option to write output and commit them > to a git repository, or we could rely on whatever's calling oe-selftest > (i.e. the autobuilder) to collect the oe-selftest output and commit it to a > repository. > > The closest analogue we currently have is buildhistory, which has logic to > push to create a local git repo of the buildhistory and push it to a remote > based on OE configuration variables. Would it make sense to do similar for > oe-selftest? I was talking to Mariano about this procedure of having a local repo then push to a remote based on config and I believe it is the best way to proceed. 68421 8 mariano.lopez 2016-11-17 18:43:20 +0000 I was working on using a git repo to store the results of the tests and is almost completed. Also in the last couple of weeks this mechanism has become more popular so, I moved the original code to an OE library. Benjamin and I also had a chat with what to do with the test metadata and we concluded that we need an XML file for this. My proposal for this file would be: <?xml version="1.0" ?> <metadata> <hostname>yctb01</hostname> <machine>qemux86-64</machine> <distro>poky</distro> <distro_version>2.2</distro_version> <layers> <meta> <branch>mariano/bug9954v2</branch> <revision>43e652f3d1fee5ce7fad67e6400315eab1b34270</revision> </meta> <meta-poky> <branch>mariano/bug9954v2</branch> <revision>43e652f3d1fee5ce7fad67e6400315eab1b34270</revision> </meta-poky> <meta-yocto-bsp> <branch>mariano/bug9954v2</branch> <revision>43e652f3d1fee5ce7fad67e6400315eab1b34270</revision> </meta-yocto-bsp> <meta-selftest> <branch>mariano/bug9954v2</branch> <revision>43e652f3d1fee5ce7fad67e6400315eab1b34270</revision> </meta-selftest> </layers> </metadata> 68432 9 markus.lehtonen 2016-11-18 06:59:39 +0000 It would be good to sync with bug 10582. There is a proposal for Git branch and tag naming, for example. Also, bug 10590 is about test report format for build performance tests Just a note: oe-build-perf-test script handles committing results to a (local) git repository but is not able to do git push. 68433 10 markus.lehtonen 2016-11-18 07:04:04 +0000 Regarding the proposed xml metadata, I think it might make sense to add some host distro information as well, e.g. NAME and VERSION from /etc/os-release. I would also include distro branch and revision information, i.e. branch and revision of poky. The <layers> element would then only list additional layers 68448 11 benjamin.esquivel 2016-11-19 00:14:51 +0000 (In reply to comment #9) > It would be good to sync with bug 10582. There is a proposal for Git branch > and tag naming, for example. > > Also, bug 10590 is about test report format for build performance tests As I discussed earlier with Markus, both performance and test results can use a common structure. > > Just a note: oe-build-perf-test script handles committing results to a > (local) git repository but is not able to do git push. 68449 12 benjamin.esquivel 2016-11-19 01:24:50 +0000 (In reply to comment #10) > Regarding the proposed xml metadata, I think it might make sense to add some > host distro information as well, e.g. NAME and VERSION from /etc/os-release. I agree to this, > > I would also include distro branch and revision information, i.e. branch and > revision of poky. The <layers> element would then only list additional layers and agree to this one too. This is a good time for putting the report template in the wiki. I'll work with Mariano so we put something that takes performance also into account. I propose putting it here: https://wiki.yoctoproject.org/wiki/QA/Test_Report_Template 68512 13 mariano.lopez 2016-11-22 20:49:52 +0000 (In reply to comment #10) > Regarding the proposed xml metadata, I think it might make sense to add some > host distro information as well, e.g. NAME and VERSION from /etc/os-release. It does, indeed > I would also include distro branch and revision information, i.e. branch and > revision of poky. The <layers> element would then only list additional layers Perhaps in our use case of poky combo layer makes sense, but for cases where the user have downloaded layers separately it would make sense to have the revision information per layer. 68527 14 benjamin.esquivel 2016-11-22 23:47:13 +0000 I have started the documentation here: https://wiki.yoctoproject.org/wiki/QA/Automated_Test_Report_Template The rest of the work is noted there, summarizing: 1. extend our JUnit-XML template to accept performance data (edit it) 2. generate a performance sample and link it there (based on the template) 3. create the execution information template (all common things for the run). Markus has already something in this regard. 68531 15 markus.lehtonen 2016-11-23 08:22:52 +0000 (In reply to comment #13) > > I would also include distro branch and revision information, i.e. branch and > > revision of poky. The <layers> element would then only list additional layers > > Perhaps in our use case of poky combo layer makes sense, but for cases where > the user have downloaded layers separately it would make sense to have the > revision information per layer. Yes, that's implicit in my proposal :) If you clone oe-core and every layer separately, you would get git revision of oe-core as the "main revision" and every separately downloaded layer as <layer> element 68682 16 mariano.lopez 2016-11-29 14:47:33 +0000 Sent patch to the mailing list: http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129399.html 68780 17 mariano.lopez 2016-12-01 16:06:57 +0000 Sent v2 to the mailing list: http://lists.openembedded.org/pipermail/openembedded-core/2016-December/129516.html 68835 18 markus.lehtonen 2016-12-05 15:23:18 +0000 I put a modified metadata format proposal onto the wiki page: https://wiki.yoctoproject.org/wiki/QA/Automated_Test_Report_Template In the proposal I added distro version information (as I suggested above) and changed the layers element only to contain additional/modified layers. I'm also thinking that we should probably add bitbake version there, too, (this information would be present in case it was customized). I'm not an XML guru so I'm not sure how to balance between xml (sub-)elements and attributes. Following the JUnit style, I used more attributes than Mariano's original proposal. In any case, I think it would be logical to have e.g. all host distro related information under one <host_distro> element (either attributes or subelements) rather than having multiple "parallel" host_distro, host_distro_version, host_distro_pretty_name etc. elements. I would probably change the "distro" tag(s) to "product" (because oe-core, for example, is not a distro as such) but I could do with "distro" as well. 69128 19 ross.burton 2016-12-14 16:36:54 +0000 Merged in oe-core 758e4b7e619f9aaa79e3910b5539ff8fb2d1064a.