Bug 10033

Summary: oe-selftest: parallelize runs
Product: [QA/Testing] Build Testing Reporter: Mariano Lopez <mariano.lopez>
Component: generalAssignee: Unassigned <unassigned>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: anibal.limon, benjamin.esquivel, joshuagloe, juan.m.cruz.alcaraz, leonardo.sandoval.gonzalez, richard.purdie
Version: unspecified   
Target Milestone: 4.99   
Hardware: x86   
OS: Multiple   
Whiteboard: LET
OS type for building Yocto: --- Type of Regression: New (Never tested)
Verified: Documentation change: Don't know
Bug Depends on: 11429    
Bug Blocks:    
Attachments:
Description Flags
oe-selftest batch run
none
oe-selftest 4 pythreads with 16 bb process/parallel make
none
oe-selftest batch run none

Description Mariano Lopez 2016-07-26 20:37:25 UTC
Currently the autobuilders take about 7 hours to execute a single run of oe-selftest, one way to improve the time is to parallelize the tests.
Comment 1 Benjamin Esquivel 2016-11-03 21:58:52 UTC
Mariano, now that you've been working on the OEQA2 effort, how do you see this coming for 2.3? can you estimate a Milestone for this feature?
Comment 2 Mariano Lopez 2016-11-03 22:18:00 UTC
I don't think this is doable in 2.3, selftest would require a lot of effort to migrate to the new framework.
Comment 3 Benjamin Esquivel 2017-02-09 16:43:26 UTC
Moving to 2.4 because this bug depends on pending work for the selftest script.
Comment 5 Aníbal Limón 2017-07-11 15:37:40 UTC
Created attachment 3862 [details]
oe-selftest batch run
Comment 6 Aníbal Limón 2017-07-11 15:38:22 UTC
Created attachment 3863 [details]
oe-selftest 4 pythreads with 16 bb process/parallel make
Comment 7 Aníbal Limón 2017-07-11 15:41:01 UTC
Comment on attachment 3862 [details]
oe-selftest batch run

Wrong log
Comment 8 Aníbal Limón 2017-07-11 15:41:53 UTC
Created attachment 3864 [details]
oe-selftest batch run
Comment 10 Leonardo Sandoval Gonzalez 2017-08-29 16:55:22 UTC
Anibal will keep working on it, but while he gets an external account, I am the temporal owner of this bug.
Comment 11 Leonardo Sandoval Gonzalez 2017-10-20 16:12:43 UTC
there is a general agreement to use different approach with the idea to make it simpler and more maintainable: Use a wrapper script for oe-selftest, which will detect test cases and call each one on a separate build directory. One we have this setup, the next point is to figure out how much can be parallelized based on test cases dependencies and system resources.
Comment 12 Leonardo Sandoval Gonzalez 2017-10-31 15:07:52 UTC
The solution proposed on bug 11429 allows oe-selftest parallelization as demonstrated in [1]. However, the proposed oe-selftest does not **any** parallelization by default (just one job) so this bug is not fully solved. The final solution is to create **another** (I am not really sure if this is the path to go) script which basically does the following

    $ echo $ALLTESTS | time parallel --jobs4 oe-selftest -r

where ALLTESTS is a variable containing all tests cases (modules) found by the
the runner (oe-selftest-internal) (i.e. ALLTESTS="$(oe-selftest -m |
awk '{ print $NF }' | grep -v ':')").


[1] http://lists.openembedded.org/pipermail/openembedded-core/2017-October/143697.html
Comment 13 Leonardo Sandoval Gonzalez 2017-12-18 17:52:15 UTC
I have submitted a patch with does the parallelization using GNU parallel [1] but in a internal discussion with RP, this is not the approach we want. Ideally, we should do the parallelization inside the meta/lib/oeqa/core as Anibal started but not concluded AFAIK.

[1] http://lists.openembedded.org/pipermail/openembedded-core/2017-December/145742.html
Comment 14 Richard Purdie 2018-09-13 15:17:58 UTC
Patches to parallelise oe-selftest merged for 2.6.