10827 2016-12-20 14:33:08 +0000 Run oe-selftest -r wic for every patch coming to openembedded-core, poky and bitbake-devel lists 2024-10-21 18:09:53 +0000 1 1 6 Yocto Project Subprojects Patchwork/Patchtest Patchtest Testsuite 2.3 x86 Multiple RESOLVED OBSOLETE LET Medium enhancement Future 1 eduard.bartosh tgamblin bluelightning leonardo.sandoval.gonzalez randy.macleod richard.purdie tgamblin --- --- Don't know oldest_to_newest 69226 0 eduard.bartosh 2016-12-20 14:33:08 +0000 This is a follow-up bug for bug 10620, which was closed as duplicate. The idea is to setup patchtest to test every patch coming to the set of mailling lists by running set of tests and send feedback to the maintainer. As a maintainer of oe image creator(wic) I'd like to run selftest --coverage -r wic and get notified if any test or test coverage regression happens. This would allow me to react on potential breakage of wic codebase much faster than I can I'm currently able to do. Wic test run currently takes around 20 minutes, which is hopefully enough to test every patch. If this setup will work we can test other software components similar way, which would be a great first step towards working continuous integration for the project. 69350 1 eduard.bartosh 2016-12-22 21:43:31 +0000 With the great help from Leo I can now use patchtest to run 'oe-selftest -r wic' this way: ./patchtest/patchtest --repo-dir ./openembedded-core/ --start-dir ./patchtest-oe/tests mbox-4508 -p '*selftest*' -j {"status": "FAIL", "results": {"issue": "The script oe-selftest fail running wic", "fix": "Make sure you run cd /home/ed/git/yocto/patchtest/openembedded-core; . /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest -r wic on top of your series", "data": [["Command", "cd /home/ed/git/yocto/patchtest/openembedded-core; . /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest -r wic"], ["Code", 1]], "id": "test_oeselftest.OESelfTest.test_wic"}} ---------------------------------------------------------------------- Ran 1 test in 159.105s OK However, it hides the output of the command, so it's quite hard for me as a maintainer to understand why test fails. 69351 2 eduard.bartosh 2016-12-22 21:45:52 +0000 I'd also like to see which patch series is under tests. And I'd like to have all this info sent to me by e-mail when test failed. 69379 3 leonardo.sandoval.gonzalez 2016-12-24 02:04:13 +0000 (In reply to comment #1) > With the great help from Leo I can now use patchtest to run 'oe-selftest -r > wic' this way: > > ./patchtest/patchtest --repo-dir ./openembedded-core/ --start-dir > ./patchtest-oe/tests mbox-4508 -p '*selftest*' -j > {"status": "FAIL", "results": {"issue": "The script oe-selftest fail running > wic", "fix": "Make sure you run cd > /home/ed/git/yocto/patchtest/openembedded-core; . > /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest > -r wic on top of your series", "data": [["Command", "cd > /home/ed/git/yocto/patchtest/openembedded-core; . > /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest > -r wic"], ["Code", 1]], "id": "test_oeselftest.OESelfTest.test_wic"}} > ---------------------------------------------------------------------- > Ran 1 test in 159.105s > > OK > > > However, it hides the output of the command, so it's quite hard for me as a > maintainer to understand why test fails. I will provide a wic branch under the patchtest-oe so we can start from that. I will include more oe-selftest logging in case of failure. 69380 4 leonardo.sandoval.gonzalez 2016-12-24 02:07:14 +0000 (In reply to comment #2) > I'd also like to see which patch series is under tests. And I'd like to have > all this info sent to me by e-mail when test failed. This part is being documented and once tested, I will place the info inside the patchtest repo. 69400 5 leonardo.sandoval.gonzalez 2016-12-28 19:59:49 +0000 (In reply to comment #1) > With the great help from Leo I can now use patchtest to run 'oe-selftest -r > wic' this way: > > ./patchtest/patchtest --repo-dir ./openembedded-core/ --start-dir > ./patchtest-oe/tests mbox-4508 -p '*selftest*' -j > {"status": "FAIL", "results": {"issue": "The script oe-selftest fail running > wic", "fix": "Make sure you run cd > /home/ed/git/yocto/patchtest/openembedded-core; . > /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest > -r wic on top of your series", "data": [["Command", "cd > /home/ed/git/yocto/patchtest/openembedded-core; . > /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest > -r wic"], ["Code", 1]], "id": "test_oeselftest.OESelfTest.test_wic"}} > ---------------------------------------------------------------------- > Ran 1 test in 159.105s > > OK > > > However, it hides the output of the command, so it's quite hard for me as a > maintainer to understand why test fails. Ed, can you fetch patchtest-oe and move to the wic branch? I have include those wic tests that have failed, at least more info that before. wic branch wont be merge into master because oe-selftest execution is at this point impossible to run at production. 69401 6 leonardo.sandoval.gonzalez 2016-12-28 20:13:20 +0000 Ed Regarding the title, for the moment the test suite and the automatic testing is focused on oe-core, not other project (like bb or poky). In case you want poky, patches coming any related project (bb, oe-core and poky itself) will fit but several scripts required this change (s/oe-core/poky), including the guest machine and the user needs to do it because the later wont be supported. In the other hand, documentation has been updated. Take a look and let me know what you think. 69431 7 eduard.bartosh 2017-01-02 18:53:11 +0000 I used wic branch, but the output is still not enough from my point of view. It doesn't give any information about the reason of the failure: (patchtest) openembedded-core (master)> time ../patchtest/patchtest - --repo-dir . --start-dir ../patchtest-oe/tests/ --pattern '*oeselftest*' --json --debug < /tmp/patch CMD: git rev-parse --abbrev-ref HEAD RCODE: 0 STDOUT: master STDERR: Reading /tmp/tmpiqwhQ8 contents CMD: git rev-parse --short HEAD RCODE: 0 STDOUT: 425afe2 STDERR: CMD: git checkout -b patchtest_9141 425afe2 RCODE: 0 STDOUT: STDERR: Switched to a new branch 'patchtest_9141' CMD: git apply --check --verbose Parameters Repository : /home/ed/git/yocto/patchtest/openembedded-core Target Commit : 425afe2 Target Branch : master Working branch : patchtest_9141 Patch : <patchtestpatch.Patch object at 0x7f7fc780b7d0> Pattern ^FAIL: not found on bitbake output {"status": "FAIL", "results": {"issue": "The script oe-selftest fail running wic", "fix": "Make sure you run cd /home/ed/git/yocto/patchtest/openembedded-core; . /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest -r wic on top of your series", "data": [["Wic Failures", ""]], "id": "test_oeselftest.OESelfTest.test_wic"}} CMD: git checkout master RCODE: 0 STDOUT: Your branch is up-to-date with 'origin/master'. STDERR: Switched to branch 'master' CMD: git branch -D patchtest_9141 RCODE: 0 STDOUT: Deleted branch patchtest_9141 (was 425afe2). STDERR: ---------------------------------------------------------------------- Ran 1 test in 477.401s OK real 7m57.519s user 5m28.080s sys 0m25.236s (patchtest) openembedded-core (master)> git --git-dir ../patchtest-oe/.git branch master * wic 69585 8 leonardo.sandoval.gonzalez 2017-01-06 17:21:43 +0000 (In reply to comment #7) > I used wic branch, but the output is still not enough from my point of view. > It doesn't give any information about the reason of the failure: Right. What I provided is a working sample code that can serve as a base for your work on oe-selftest/wic. Feel free to modify it and when something is ready for pushing to upstream, I can review it and merge it. 70065 9 eduard.bartosh 2017-01-23 23:12:22 +0000 Moving to M3. It requires more time to work on. 70909 10 eduard.bartosh 2017-02-23 15:56:14 +0000 moving to 2.4 99962 11 tgamblin 2024-10-21 18:09:53 +0000 I don't think this issue is still relevant to the current rendition of patchtest. We can reopen it if that changes.