Bug 9080

Summary: Autobuilder/oe-selftest: Have a way to specify a which oe-self tests get run on a forced AB job
Product: [Infrastructure] AutoBuilder Reporter: Benjamin Esquivel <benjamin.esquivel>
Component: autobuilderAssignee: Unassigned <unassigned>
Status: RESOLVED WONTFIX QA Contact: Apoorv <apoorv.sangal>
Severity: enhancement    
Priority: Medium CC: akuster, alexandru.c.georgescu, anibal.limon, benjamin.esquivel, brianbrowns123, elizabeth.flanagan, infras.ab.watcher, Infras.watcher, jhonmerced5, jose.perez.carranza, joshuagloe, leonardo.sandoval.gonzalez, randy.macleod, richard.purdie
Version: unspecified   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Benjamin Esquivel 2016-02-09 10:16:57 UTC
Currently, you have no easy way of running a specified set of oe-selftest at a forced build of the autobuilder. So you have to wait hours to see on-demand tests.
Comment 1 Beth Flanagan 2016-02-11 16:07:43 UTC
What are you trying to accomplish here? 

If you want to do a subset of the tests, if you can give me a list of those subsets (and the default) we can do a select box for them. We do not allow generally a field where you can just input <image> and bitbake will run it (there are security concerns in doing so).
Comment 2 Benjamin Esquivel 2016-02-11 16:20:27 UTC
What you describe is fine, a way of selecting one or more tests to be run (test granularity is better, could be grouped by modules). The tests might come as a dynamic list that you need to read for each of the builds you want to run into (listing via oe-selftest --list-tests or something along the params)

A feature like this would make on-demand builds to be way faster than the added 3 to 5 hours oe-selftest imposes.
Comment 3 Beth Flanagan 2016-02-11 16:38:30 UTC
Well, it can't really be done as a dynamic list as the autobuilder has no knowledge as to what is in the codebase until Force Build is pressed and by that time, it's too late to tell it what to run.
Comment 4 Benjamin Esquivel 2016-02-11 16:44:18 UTC
Okay, understand that now. We could try to write a static list of tests now and it would still be useful in most of the cases.

Though, the ideal would be that you can select what's available.

Do you think we could achieve that at some point?
Comment 5 Beth Flanagan 2016-02-11 16:49:23 UTC
Probably not in the way you want it. I could make it so that we keep a matrix of branch/tests in the autobuilder code base, but that wouldn't be particularly useful I don't think.

One way we could maybe do it is have an abstracted out set of tests that are static on the autobuilder, but change within oe?
Comment 6 Benjamin Esquivel 2016-02-11 19:09:11 UTC
That would be a good first approach at least, how would we specify the tests to run perhaps is the first question.
Comment 7 Leonardo Sandoval Gonzalez 2016-05-25 20:44:48 UTC
Due to limited resources, moving this to 2.3
Comment 8 Leonardo Sandoval Gonzalez 2017-01-25 16:32:51 UTC
Anibal, any further done on this area during M2?

M2 is over so moving to M3.
Comment 9 Aníbal Limón 2017-04-20 15:16:27 UTC
Moving to 2.4 because is an enhancement.
Comment 10 Jhon Merced 2017-11-13 14:50:58 UTC
Neverending moving? ;) Hope to see it implemented soon.
https://www.premiumlinkgenerator.com
Comment 11 brianbrowns 2018-01-15 16:21:53 UTC
Hi guys, what's the status here? I'm waiting too. https://promocode.com.ph/
Comment 12 Armin Kuster 2019-12-23 16:37:53 UTC
With the new AB, is this still an issue?
Comment 13 Randy MacLeod 2021-12-23 15:51:11 UTC
Complicated to add to AB and we don't really see the need.