Bug 10827

Summary:	Run oe-selftest -r wic for every patch coming to openembedded-core, poky and bitbake-devel lists
Product:	[Yocto Project Subprojects] Patchwork/Patchtest	Reporter:	Ed Bartosh <eduard.bartosh>
Component:	Patchtest Testsuite	Assignee:	Trevor Gamblin <tgamblin>
Status:	RESOLVED OBSOLETE	QA Contact:
Severity:	enhancement
Priority:	Medium	CC:	bluelightning, leonardo.sandoval.gonzalez, randy.macleod, richard.purdie, tgamblin
Version:	2.3
Target Milestone:	Future
Hardware:	x86
OS:	Multiple
Whiteboard:	LET
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description Ed Bartosh 2016-12-20 14:33:08 UTC

This is a follow-up bug for bug 10620, which was closed as duplicate.

The idea is to setup patchtest to test every patch coming to the set of mailling lists by running set of tests and send feedback to the maintainer.

As a maintainer of oe image creator(wic) I'd like to run selftest --coverage -r wic and get notified if any test or test coverage regression happens.

This would allow me to react on potential breakage of wic codebase much faster than I can I'm currently able to do.

Wic test run currently takes around 20 minutes, which is hopefully enough to test every patch.

If this setup will work we can test other software components similar way, which would be a great first step towards working continuous integration for the project.

Comment 1 Ed Bartosh 2016-12-22 21:43:31 UTC

With the great help from Leo I can now use patchtest to run 'oe-selftest -r wic' this way:

./patchtest/patchtest --repo-dir ./openembedded-core/ --start-dir ./patchtest-oe/tests mbox-4508 -p '*selftest*' -j
{"status": "FAIL", "results": {"issue": "The script oe-selftest fail running wic", "fix": "Make sure you run cd /home/ed/git/yocto/patchtest/openembedded-core; . /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest -r wic on top of your series", "data": [["Command", "cd /home/ed/git/yocto/patchtest/openembedded-core; . /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest -r wic"], ["Code", 1]], "id": "test_oeselftest.OESelfTest.test_wic"}}
----------------------------------------------------------------------
Ran 1 test in 159.105s

OK


However, it hides the output of the command, so it's quite hard for me as a maintainer to understand why test fails.

Comment 2 Ed Bartosh 2016-12-22 21:45:52 UTC

I'd also like to see which patch series is under tests. And I'd like to have all this info sent to me by e-mail when test failed.

Comment 3 Leonardo Sandoval Gonzalez 2016-12-24 02:04:13 UTC

(In reply to comment #1)
> With the great help from Leo I can now use patchtest to run 'oe-selftest -r
> wic' this way:
> 
> ./patchtest/patchtest --repo-dir ./openembedded-core/ --start-dir
> ./patchtest-oe/tests mbox-4508 -p '*selftest*' -j
> {"status": "FAIL", "results": {"issue": "The script oe-selftest fail running
> wic", "fix": "Make sure you run cd
> /home/ed/git/yocto/patchtest/openembedded-core; .
> /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest
> -r wic on top of your series", "data": [["Command", "cd
> /home/ed/git/yocto/patchtest/openembedded-core; .
> /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest
> -r wic"], ["Code", 1]], "id": "test_oeselftest.OESelfTest.test_wic"}}
> ----------------------------------------------------------------------
> Ran 1 test in 159.105s
> 
> OK
> 
> 
> However, it hides the output of the command, so it's quite hard for me as a
> maintainer to understand why test fails.

I will provide a wic  branch under the patchtest-oe so we can start from that. I will include more oe-selftest logging in case of failure.

Comment 4 Leonardo Sandoval Gonzalez 2016-12-24 02:07:14 UTC

(In reply to comment #2)
> I'd also like to see which patch series is under tests. And I'd like to have
> all this info sent to me by e-mail when test failed.

This part is being documented and once tested, I will place the info inside the patchtest repo.

Comment 5 Leonardo Sandoval Gonzalez 2016-12-28 19:59:49 UTC

(In reply to comment #1)
> With the great help from Leo I can now use patchtest to run 'oe-selftest -r
> wic' this way:
> 
> ./patchtest/patchtest --repo-dir ./openembedded-core/ --start-dir
> ./patchtest-oe/tests mbox-4508 -p '*selftest*' -j
> {"status": "FAIL", "results": {"issue": "The script oe-selftest fail running
> wic", "fix": "Make sure you run cd
> /home/ed/git/yocto/patchtest/openembedded-core; .
> /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest
> -r wic on top of your series", "data": [["Command", "cd
> /home/ed/git/yocto/patchtest/openembedded-core; .
> /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest
> -r wic"], ["Code", 1]], "id": "test_oeselftest.OESelfTest.test_wic"}}
> ----------------------------------------------------------------------
> Ran 1 test in 159.105s
> 
> OK
> 
> 
> However, it hides the output of the command, so it's quite hard for me as a
> maintainer to understand why test fails.

Ed, can you fetch patchtest-oe and move to the wic branch? I have include those wic tests that have failed, at least more info that before. wic branch wont be merge into master because oe-selftest execution is at this point impossible to run at production.

Comment 6 Leonardo Sandoval Gonzalez 2016-12-28 20:13:20 UTC

Ed

Regarding the title, for the moment the test suite and the automatic testing is focused on oe-core, not other project (like bb or poky). In case you want poky, patches coming any related project (bb, oe-core and poky itself) will fit but several scripts required this change (s/oe-core/poky), including the guest machine and the user needs to do it because the later wont be supported.

In the other hand, documentation has been updated. Take a look and let me know what you think.

Comment 7 Ed Bartosh 2017-01-02 18:53:11 UTC

I used wic branch, but the output is still not enough from my point of view. It doesn't give any information about the reason of the failure:

(patchtest) openembedded-core (master)> time ../patchtest/patchtest - --repo-dir . --start-dir ../patchtest-oe/tests/ --pattern '*oeselftest*' --json --debug < /tmp/patch 
CMD: git rev-parse --abbrev-ref HEAD RCODE: 0 STDOUT: master STDERR: 
Reading /tmp/tmpiqwhQ8 contents
CMD: git rev-parse --short HEAD RCODE: 0 STDOUT: 425afe2 STDERR: 
CMD: git checkout -b patchtest_9141 425afe2 RCODE: 0 STDOUT:  STDERR: Switched to a new branch 'patchtest_9141'
CMD: git apply --check --verbose
Parameters
	Repository     : /home/ed/git/yocto/patchtest/openembedded-core
	Target Commit    : 425afe2
	Target Branch    : master
	Working branch : patchtest_9141
	Patch          : <patchtestpatch.Patch object at 0x7f7fc780b7d0>
Pattern ^FAIL: not found on bitbake output
{"status": "FAIL", "results": {"issue": "The script oe-selftest fail running wic", "fix": "Make sure you run cd /home/ed/git/yocto/patchtest/openembedded-core; . /home/ed/git/yocto/patchtest/openembedded-core/oe-init-build-env;oe-selftest -r wic on top of your series", "data": [["Wic Failures", ""]], "id": "test_oeselftest.OESelfTest.test_wic"}}
CMD: git checkout master RCODE: 0 STDOUT: Your branch is up-to-date with 'origin/master'. STDERR: Switched to branch 'master'
CMD: git branch -D patchtest_9141 RCODE: 0 STDOUT: Deleted branch patchtest_9141 (was 425afe2). STDERR: 
----------------------------------------------------------------------
Ran 1 test in 477.401s

OK

real	7m57.519s
user	5m28.080s
sys	0m25.236s

(patchtest) openembedded-core (master)> git --git-dir ../patchtest-oe/.git branch
  master
* wic

Comment 8 Leonardo Sandoval Gonzalez 2017-01-06 17:21:43 UTC

(In reply to comment #7)
> I used wic branch, but the output is still not enough from my point of view.
> It doesn't give any information about the reason of the failure:

Right. What I provided is a working sample code that can serve as a base for your work on oe-selftest/wic. Feel free to modify it and when something is ready for pushing to upstream, I can review it and merge it.

Comment 9 Ed Bartosh 2017-01-23 23:12:22 UTC

Moving to M3. It requires more time to work on.

Comment 10 Ed Bartosh 2017-02-23 15:56:14 UTC

moving to 2.4

Comment 11 Trevor Gamblin 2024-10-21 18:09:53 UTC

I don't think this issue is still relevant to the current rendition of patchtest. We can reopen it if that changes.