Bug 8548 - Document best practises/style guide for selftest tests and update tests
Summary: Document best practises/style guide for selftest tests and update tests
Status: RESOLVED FIXED
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 5.99
Hardware: All Multiple
: Medium enhancement
Target Milestone: 3.1.2
Assignee: Mark Morton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-10-20 14:18 UTC by Ross Burton
Modified: 2020-06-19 13:07 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ross Burton 2015-10-20 14:18:46 UTC
The selftest code has evolved over time and now has some interesting quirks and inconsistent behaviour.  A canonical description of how to write a test needs to be written and the test framework/suite updated.
Comment 1 Joshua Lock 2017-03-13 21:27:19 UTC
Assigning to Aníbal, CC Mariano. These two have rewritten OEQA and are better positioned to document how to write a test.

Hopefully the updated tests are already more consistent?

Aníbal/Mariano — feel free to ask for review of the docs here.
Comment 2 Joshua Lock 2017-03-28 09:21:57 UTC
As part of this we should include a list of do/don't items for selftests. One that came up recently[1] is that we really need to be careful to ensure that the tests are idempotent. We share the DL_DIR and SSTATE_CACHE across the entire autobuilder so anything that wipes files from those shared directories should be avoided.

1. http://git.yoctoproject.org/clean/cgit.cgi/poky/commit/?id=58a02e73d0a6b110d96e94033e1fdf634b42ccec
Comment 3 Aníbal Limón 2017-05-25 20:59:37 UTC
Leo and myself are finishing the oe-selftest migration into the new framework, after that, we will write a document describing best practices to write tests for runtime, sdk, esdk and selftest.

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/oeqa_integration

Cheers,
Anibal
Comment 4 Leonardo Sandoval Gonzalez 2017-06-01 15:45:46 UTC
With the new oe-qa framework about to be merged into master during M1, moving this into M2 and expected to be finished early 2.4 M2.
Comment 5 Aníbal Limón 2017-06-01 16:02:48 UTC
Since the main users of the OEQA framework is the QA team, it is the best to someone of them document it.
Comment 6 Aníbal Limón 2017-06-13 15:04:08 UTC
(In reply to comment #5)
> Since the main users of the OEQA framework is the QA team, it is the best to
> someone of them document it.

Some part that need to be added is about migration to the new QA framework,

- oe-selftest cases needs to execute setUpClass to create it's own builddir.
Comment 7 Mikko Ylinen 2017-06-13 15:11:06 UTC
The BKM/lessons learnt from the refkit team to use v2:

 * moved the files to <layer>/lib/oeqa/selftest/cases/
 * updated class imports with (we're not using decorators):

       sed -i -e '/decorators/d; \
                 s/oeSelfTest/OESelftestTestCase/g; \
                 s/selftest\.base/selftest\.case/; \
                 s/selftest\.sstate/selftest\.cases\.sstate/'

 * changed filenames with '-' in them to have '_' instead due to how
    python unittest loader sees (or does not) the files.
 * added modules using SetUpClass() to call their parent's SetUpClass()
 * changed self.log to self.logger

(the sed part is refkit specific but I noticed some parts of the transition is easy to automate/script. it worked with 'find meta-refkit* -wholename '*selftest/cases/*.py'|sed ...' just OK)
Comment 8 Libertad 2017-06-29 17:33:28 UTC
As I understand this in this bug the idea is to better describe the Basic architecture information of the oeqa test. On this wiki https://wiki.yoctoproject.org/wiki/Oe-selftest. Describe how a new person can contribute to creating testcases for their new modules.

Make documentation proposal on this framework that later we can decide to add it to the mega manual or some other place. We will decide upon the content. 

The core development of this framework under 
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/core

Start by reading the README then the context.py

http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html
Comment 9 Armin Kuster 2019-12-22 18:37:10 UTC
we should still do this
Comment 10 Mark Morton 2020-06-19 13:07:11 UTC
I completed the first version of the new test-manual source and added it to the yocto-docs repo. There are a few minor updates, but the document itself is complete.