8648 2015-11-05 10:19:13 +0000 Automatic testing of incoming patches 2017-12-06 16:49:51 +0000 1 1 6 Yocto Project Subprojects Patchwork/Patchtest Patchwork unspecified All Multiple RESOLVED FIXED Testing in Progress High enhancement 2.2 M2 7655 7658 9130 8714 4078 8649 8716 9244 1 bluelightning leonardo.sandoval.gonzalez benjamin.esquivel jku leonardo.sandoval.gonzalez mariano.lopez mhalstead randy.e.witt richard.purdie ross.burton sgw --- --- Don't know oldest_to_newest 56218 0 bluelightning 2015-11-05 10:19:13 +0000 We need a mechanism triggered from patchwork to take a patch/patchset submitted to the mailing list, apply it to the appropriate branch (usually master), run appropriate tests (see bug 4078, bug 6370, bug 8647), email the results to the mailing list CCing the submitter and if appropriate record failure status in patchwork (see bug 7658). (This isn't so much of an umbrella bug, rather covering the glue script(s) and setup we'll need to make it all work. The full implementation will need the functionality described in the bugs this one depends on, but a basic implementation can be started in advance of having that.) 56606 1 2856 leonardo.sandoval.gonzalez 2015-11-18 23:05:55 +0000 Created attachment 2856 Patch workflow between submitter/maintainer and patchwork This is a proposed workflow for merging and testing patches coming from the mailing list. The introduction of patch's tags are important to keep track of the patch status. This workflow takes into account the current bug and 8649 [1] [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8649 56665 2 randy.e.witt 2015-11-19 18:33:53 +0000 Leo, I would highly recommend using containers to do the tests against the patch. I will shortly be sending some instructions to other GDC guys with some container testing I had setup, you may be able to leverage that somehow. If you have questions, let me know. 56669 3 leonardo.sandoval.gonzalez 2015-11-19 21:46:58 +0000 (In reply to comment #2) > Leo, I would highly recommend using containers to do the tests against the > patch. I will shortly be sending some instructions to other GDC guys with > some container testing I had setup, you may be able to leverage that somehow. > Thanks Randy. Looking forward to see the instructions. > If you have questions, let me know. 56763 4 leonardo.sandoval.gonzalez 2015-11-24 23:08:07 +0000 I have found this useful tool call git-pw [1]. I have not fully used the tool, but apparently for testing new patchwork patches into a repository, it would be a matter of using this tool with apply and apply-patch parameters. It is interesting to notice that this tool is part of patchwork repository [2] maintained by Damien Lespiau; [1] http://patchwork-freedesktop.readthedocs.org/en/latest/manual.html#git-pw [2] https://github.com/dlespiau/patchwork 56767 5 bluelightning 2015-11-25 00:47:35 +0000 git-pw could definitely be useful; an interesting challenge we have though is that we often receive (and encourage submitters to send) pull requests which come with a git branch, and it's fairly common practice for that branch to be updated without sending a new patch when making a minor change (such as fixing a typo or commit message); AIUI Ross and Richard tend to grab the branch rather than the individual patches because it's easier, so if they differ from the patches posted to the list that could be an issue. It may be that the simplest way of handling this is to ban the practice of updating the git branch without reposting a new version of at least the affected patch. We have received complaints in the past from at least one community member about branches not matching with the posted patches. However, even if we did this we'd have to have a check somewhere to verify that the patches do in fact correspond to the branch. (This is perhaps something to consider as a separate work task, but we do need to keep it in mind.) 56785 6 benjamin.esquivel 2015-11-25 19:52:55 +0000 Paul, Leonardo, as you know, the nature of branch:HEADs is volatile, when you send a branch for review and later you update such branch then the review material gets outdated automatically. I've seen in the past that working around such behavior of the branches could be done via specifying branch:sha or using git tags. Both of them with pros/cons and involves changes in the process that are not easily taken by the community as they usually mean extra work. It could be proposed that with the merge-request that a developer sends, it includes the sha or it gets tagged (bug number perhaps?) so the reference does not move if the branch gets updated. 57179 7 leonardo.sandoval.gonzalez 2015-12-10 17:48:16 +0000 I missed M1. This feature will be tied to the git-pw tool, tool present on the damien's PW repo. Some sub-commands are missing on this tool, thus focusing on it, specially [1] and [2]. [1] https://github.com/dlespiau/patchwork/issues/115 [2] https://github.com/dlespiau/patchwork/issues/113 Some scenarios using this tool: # Test if 'last' (the last series sent to the ML) series apply cleanly > git pw list --since XXX | get_id | head -1 | xargs git pw apply # Test all series since a specific date > git pw list --since XXX | get_id | xargs git pw apply # Execute some static tests on mboxes since a specific date > git pw list --since XXX | get_id | xargs git pw mbox | checkpatch.pl 58301 8 leonardo.sandoval.gonzalez 2016-01-28 15:34:20 +0000 M2 missed. Most time spent on the new tool to test patches called 'patchtest' [1]. The tool is ready to be used, and currently being tested. patchtest is just a test framework and it is not intended to include test cases/units. So all dependencies bugs should be done in a separate repo and this effort is being done under [2] repository at the moment. [1] https://github.com/lsandoval/patchtest [2] https://github.com/aphran/pt-suites 59703 9 leonardo.sandoval.gonzalez 2016-03-03 19:45:52 +0000 This enhancement was not completed on the enhancement period. Effort done by Daniel but still not fully complete. Work will be taken again on the 2.2 period. 59771 10 leonardo.sandoval.gonzalez 2016-03-07 22:12:14 +0000 By mistake, I moved to it to 2.2 but it should be 2.1 M4, as set now. The last comment I was referring to [1] [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8647 59901 11 leonardo.sandoval.gonzalez 2016-03-11 15:56:41 +0000 Most of the desired features are now supported on [1] but testing on [2] is still pending. At least moving to a more realistic status. [1] https://github.com/lsandoval/patchtest [2] http://patchwork.openembedded.org/ 60984 12 leonardo.sandoval.gonzalez 2016-04-08 17:29:07 +0000 Michael, please assign this bug back to me once the PW instance is ready. Patchtest is practically ready to query/post the new instance. 62131 13 mhalstead 2016-05-11 21:04:43 +0000 Waiting on Patchwork upgrade. 62203 14 mhalstead 2016-05-13 22:45:38 +0000 I wanted to clarify that as far as I understand this bug is blocked by bug 9130 which I am primarily responsible for. Jose Lamego and Paul Eggleton are also working on that bug with me. Once we are finished upgrading Patchwork Leonardo Sandoval Gonzalez will be able to continue with this bug. 63406 15 leonardo.sandoval.gonzalez 2016-06-28 15:34:31 +0000 Now that new instance is up, I need to test fetching series and posting results to it, so taking the bug again. 63671 16 leonardo.sandoval.gonzalez 2016-07-06 14:03:05 +0000 patchtest system has been implemented but not fully tested on 'live' patchwork series events. The reason to move it to M2 is to have this window to test incoming patches using patchest-oe as a test suite (and patchtest as an engine). Test results wont be posted to patchwork; instead, the behavior of the system will be monitored and if necessary, changes will be done. This way, we can do changes into the system and developer does not receive any email on behalf of patchtest. Repository is still not public. 63843 17 leonardo.sandoval.gonzalez 2016-07-12 15:28:38 +0000 On the testing phase with new series and new checks located on the suite. Repository is still not public. 64071 18 leonardo.sandoval.gonzalez 2016-07-15 21:36:18 +0000 + Framework is complete (still not public). + Accepts a single patch (fetched from patchwork or created with git-format-patch) and expects a test suite to be executed. + The related test suite to be used is being tracked on https://bugzilla.yoctoproject.org/show_bug.cgi?id=4078 + Current command line arguments: > patchtest --help usage: patchtest [-h] [-C REPODIR] [--test-name TESTNAME] [--keep-branch | --no-apply] [--test-dir TESTDIR] [--branch BRANCH] [--commit COMMIT] [--debug] [--failures] [--raw] PATCH positional arguments: PATCH The patch to be tested by patchtest optional arguments: -h, --help show this help message and exit -C REPODIR Name of the repository where testing is done --test-name TESTNAME Test name to be used if results are posted. In case all items failed merged, then the test name is <--test-name>-merge-failure --keep-branch Keep the working branch after patchtest execution --no-apply Do not apply the patch into the repo --test-dir TESTDIR Directory where tests are located --branch BRANCH, -b BRANCH Branch name used by patchtest to branch from. By default, it uses the current one. --commit COMMIT, -c COMMIT Commit ID used by patchtest to branch from. By default, it uses HEAD. --debug, -d Enable debug output --failures, -f Print just failures --raw, -r Print errors coming from the test suite 64075 19 bluelightning 2016-07-15 22:41:47 +0000 I'm sorry, but as we discussed before this cannot be marked resolved until we're actually testing incoming patches, and we aren't yet doing that. 64127 20 leonardo.sandoval.gonzalez 2016-07-18 20:32:23 +0000 (In reply to comment #19) > I'm sorry, but as we discussed before this cannot be marked resolved until > we're actually testing incoming patches, and we aren't yet doing that. ok. patchtest is working now but you have not seen the emails yet produced by patchtest. I will forward these to you until M2 release. For the moment, I am using an internal patchwork server and posting goes to a particular series (I am the author) no matter what the series is being tested, so I am getting all emails for the moment. 64132 21 sgw 2016-07-18 20:45:25 +0000 Being reviewed currently 64751 22 leonardo.sandoval.gonzalez 2016-08-01 19:30:41 +0000 + Patchtest and Patchtest-oe code was reviewed by Paul Eggleton. All observations covered except https://bugzilla.yoctoproject.org/show_bug.cgi?id=10049. + Live testing is still pending but is now tracked on a separated bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10057 + A full run covering series from an internal patchwork instance was done and emails (sent by patchwork on behalf of patchtest) sent to Paul E. and myself. Summary looks good but 10049 is still pending. Several false positives checks found and these are currently being corrected. All this work is related to bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9244. + Future issues regarding patchtest (testing framework) or patchtest-oe (patch test suite) should be reported in separately. 2856 2015-11-18 23:05:55 +0000 2015-11-18 23:05:55 +0000 Patch workflow between submitter/maintainer and patchwork automatic-testing-8648.txt text/plain 3108 leonardo.sandoval.gonzalez UmVsYXRlZCBidWdzCgpCdWcgODY0OCAtIEF1dG9tYXRpYyB0ZXN0aW5nIG9mIGluY29taW5nIHBh dGNoZXMKICAgIEJ1ZyA0MDc4IC0gQSBtZXRob2QgdG8gdmFsaWRhdGUgdGhlIGNvbnRlbnRzIG9m IGNoYW5nZXMgdG8gcmVjaXBlcywgY29uZmlndXJhdGlvbiBmaWxlcyBhbmQgY2xhc3NlcyAKICAg IEJ1ZyA2MzcwIC0gUUEvb2Utc2VsZnRlc3Q6IGFkZCB0ZXN0IHN1aXRlIGZvciByZWNpcGUgdGVz dGluZyAKICAgIEJ1ZyA4NjQ3IC0gU2NyaXB0IHRvIGFuYWx5c2Ugd2hhdCBhcmVhcyBhIHBhdGNo IGNoYW5nZXMgYW5kIHdoYXQgdGVzdHMgbmVlZCBydW5uaW5nIAoKQnVnIDg2NDkgLSBTY3JpcHQg dG8gYXNzZW1ibGUgcGF0Y2hlcyBmb3IgYXV0b2J1aWxkZXIgdGVzdGluZwoKCi0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tClN1Ym1pdHRlcgkJCSAgICAgICAJICAgICBQYXRjaHdv cmsJCQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpCdWcgODY0OAkJCSAgICAg ICB8CQoJCQkJICAgICAgIHwKMS4gU3VibWl0dGVyIHNlbmRzIHBhdGNoL3BhdGNoZXMgIC0tLT4g fAogICB0byB0aGUgbWFpbGluZyBsaXN0ICAgICAgICAgICAgICAgICB8CgkJCQkgICAgICAgfAoJ CQkJICAgICAgIHwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAtLS0g MS4gVGFrZSB0aGUgc3VibWl0dGVyJ3MgcGF0Y2gvcGF0Y2hlcy4gCgkJCQkgICAgICAgfAogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0tLSAyLiBPbiBhIHRlbXBvcmFs IGJyYW5jaCwgY3JlYXRlZCBmcm9tIG1hc3RlciwgIG1lcmdlIHRoZSBwYXRjaC9wYXRjaGVzLiBJ ZiBmYWlsZWQsIAoJCQkJICAgICAgIHwgICAgIHJlcG9ydCBpdCB0byB0aGUgc3VibWl0dGVyLCB0 YWcgaXQgYXMgRkFJTF8xX01FUkdFIGFuZCBwcm9jZXNzIGZpbmlzaGVzLiBPdGhlcndpc2UgdGFn IGl0IGFzIFBBU1NfMV9NRVJHRQoyLiBTdWJtaXR0ZXIgR2V0cyBub3RpZmljYXRpb24gICAgPC0t LSB8CiAgIGZvciBtZXJnZSBmYWlsdXJlIHVuZGVyICAgICAgICAgICAgIHwKICAgY3VycmVudCBt YXN0ZXIgICAgICAgICAgICAgICAgICAgICAgfAoJCQkJICAgICAgIHwKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgfCAtLS0gMy4gRXhlY3V0ZSBhIHNldCBvZiB0ZXN0IGNh c2VzLiBTbyBmYXIsIHRoZSBmb2xsb3dpbmcgdGVzdHMgYXJlIGNvbnNpZGVyZWQKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgIGJ1dCBpdCBuZWVkcyB0byBl c2NhbGF0ZSBmb3IgZnV0dXJlIHRlc3RzLgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB8IC0tLSAgICAgICAgIFAzLjEgU3ludGF4IHZhbGlkYXRpb24gKDQwNzgpLiBUYWcg aXQgYXMgUEFTU3xGQUlMX1NZTlRBWF9URVNUCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwgLS0tICAgICAgICAgUDMuMiBCdWlsZCB2YWxpZGF0aW9uICg2MzcwKS4gVGFn IGl0IGFzIFBBU1N8RkFJTF9CVUlMRF9URVNUCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwgLS0tICAgICAgICAgUDMuMyBUZXN0cyBjcmVhdGlvbiAoODY0NykuIFRhZyBp dCBhcyBQQVNTfEZBSUxfVEVTVFNDUkVBVElPTl9URVNUCgkJCQkgICAgICAgfAoJCQkJICAgICAg IHwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAtLS0gNC4gUmVjb3Jk IHBhc3MvZmFpbCByZXN1bHRzICg3NjU4KSAodGFncz8pLiBJZiBhbnkgdGVzdCBmYWlsZWQsIHJl cG9ydGVkIHRvIHRoZSBzdWJtaXR0ZXIsCgkJCQkgICAgICAgfCAgICAgICAgIG90aGVyd2lzZSwg dGFnIHBhdGNoIGFzIFBBU1NfQUxMX1RFU1RTLgo0LiBTdWJtaXR0ZXIgZ2V0cyBub3RpZmljYXRp b24gICAgPC0tLSB8CiAgIGZvciB0ZXN0cyBmYWlsdXJlCQkgICAgICAgfAoJCQkJICAgICAgIHwK CQkJCSAgICAgICB8Ci0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCkJ1ZyA4NjQ5 CQkJICAgICAgIHwJCgkJCQkgICAgICAgfAoJCQkJICAgICAgIHwgLS0tIFA1LiBPbiB0aGUgdGVt cG9yYWwgYnJhbmNoICdtYXN0ZXItcGFzcy10ZXN0JywgbWVyZ2UgdGhlIHBhdGNoL3BhdGNoZXMu CgkJCQkgICAgICAgfCAgICAgICAgIG1hc3Rlci1wYXNzLXRlc3QgaXMgYSBicmFuY2ggY3JlYXRl ZCBmb3IgbWVyZ2luZyBwcmV2aW91cyBwYXRjaGVzIHRhZ2dlZCBhcyBQQVNTX0FMTF9URVNUUwoJ CQkJICAgICAgIHwKCQkJCSAgICAgICB8CgkJCQkgICAgICAgfCAtLS0gUDYuIElmIFA1IGZhaWxl ZCwgdGFnIGl0IGFzIEZBSUxfMl9NRVJHRSwgbm90aWZ5IG1haW50YWluZXIuIE90aGVyd2lzZSB0 YWcgaXQgYXMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAg IFBBU1NfMl9NRVJHRQoJCQkJICAgICAgIHwKMy4gTWFpbnRhaW5lciBnZXRzIG5vdGlmaWNhdGlv biAgIDwtLS0gfAogICBmb3IgbWVyZ2VkIGZhaWx1cmUgdW5kZXIgIAkgICAgICAgfAogICBtYXN0 ZXItcGFzcy10ZXN0CQkgICAgICAgfAoJCQkJICAgICAgIHwJCQoJCQkJICAgICAgIHwKLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0K