<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>8548</bug_id>
          
          <creation_ts>2015-10-20 14:18:46 +0000</creation_ts>
          <short_desc>Document best practises/style guide for selftest tests and update tests</short_desc>
          <delta_ts>2020-06-19 13:07:11 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>10</classification_id>
          <classification>QA/Testing</classification>
          <product>Runtime Testing</product>
          <component>general</component>
          <version>5.99</version>
          <rep_platform>All</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium</priority>
          <bug_severity>enhancement</bug_severity>
          <target_milestone>3.1.2</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Ross Burton">ross.burton</reporter>
          <assigned_to name="Mark Morton">mark.morton</assigned_to>
          <cc>akuster</cc>
    
    <cc>alexandru.c.georgescu</cc>
    
    <cc>anibal.limon</cc>
    
    <cc>jose.perez.carranza</cc>
    
    <cc>joshuagloe</cc>
    
    <cc>leonardo.sandoval.gonzalez</cc>
    
    <cc>libertad.gonzalez.de.la.cruz</cc>
    
    <cc>mariano.lopez</cc>
    
    <cc>randy.macleod</cc>
    
    <cc>stephano</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>55510</commentid>
    <comment_count>0</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2015-10-20 14:18:46 +0000</bug_when>
    <thetext>The selftest code has evolved over time and now has some interesting quirks and inconsistent behaviour.  A canonical description of how to write a test needs to be written and the test framework/suite updated.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>71235</commentid>
    <comment_count>1</comment_count>
    <who name="Joshua Lock">joshuagloe</who>
    <bug_when>2017-03-13 21:27:19 +0000</bug_when>
    <thetext>Assigning to Aníbal, CC Mariano. These two have rewritten OEQA and are better positioned to document how to write a test.

Hopefully the updated tests are already more consistent?

Aníbal/Mariano — feel free to ask for review of the docs here.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>71799</commentid>
    <comment_count>2</comment_count>
    <who name="Joshua Lock">joshuagloe</who>
    <bug_when>2017-03-28 09:21:57 +0000</bug_when>
    <thetext>As part of this we should include a list of do/don&apos;t items for selftests. One that came up recently[1] is that we really need to be careful to ensure that the tests are idempotent. We share the DL_DIR and SSTATE_CACHE across the entire autobuilder so anything that wipes files from those shared directories should be avoided.

1. http://git.yoctoproject.org/clean/cgit.cgi/poky/commit/?id=58a02e73d0a6b110d96e94033e1fdf634b42ccec</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>73519</commentid>
    <comment_count>3</comment_count>
    <who name="Aníbal Limón">anibal.limon</who>
    <bug_when>2017-05-25 20:59:37 +0000</bug_when>
    <thetext>Leo and myself are finishing the oe-selftest migration into the new framework, after that, we will write a document describing best practices to write tests for runtime, sdk, esdk and selftest.

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/oeqa_integration

Cheers,
Anibal</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>73612</commentid>
    <comment_count>4</comment_count>
    <who name="Leonardo Sandoval Gonzalez">leonardo.sandoval.gonzalez</who>
    <bug_when>2017-06-01 15:45:46 +0000</bug_when>
    <thetext>With the new oe-qa framework about to be merged into master during M1, moving this into M2 and expected to be finished early 2.4 M2.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>73621</commentid>
    <comment_count>5</comment_count>
    <who name="Aníbal Limón">anibal.limon</who>
    <bug_when>2017-06-01 16:02:48 +0000</bug_when>
    <thetext>Since the main users of the OEQA framework is the QA team, it is the best to someone of them document it.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>74076</commentid>
    <comment_count>6</comment_count>
    <who name="Aníbal Limón">anibal.limon</who>
    <bug_when>2017-06-13 15:04:08 +0000</bug_when>
    <thetext>(In reply to comment #5)
&gt; Since the main users of the OEQA framework is the QA team, it is the best to
&gt; someone of them document it.

Some part that need to be added is about migration to the new QA framework,

- oe-selftest cases needs to execute setUpClass to create it&apos;s own builddir.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>74077</commentid>
    <comment_count>7</comment_count>
    <who name="Mikko Ylinen">mikko.ylinen</who>
    <bug_when>2017-06-13 15:11:06 +0000</bug_when>
    <thetext>The BKM/lessons learnt from the refkit team to use v2:

 * moved the files to &lt;layer&gt;/lib/oeqa/selftest/cases/
 * updated class imports with (we&apos;re not using decorators):

       sed -i -e &apos;/decorators/d; \
                 s/oeSelfTest/OESelftestTestCase/g; \
                 s/selftest\.base/selftest\.case/; \
                 s/selftest\.sstate/selftest\.cases\.sstate/&apos;

 * changed filenames with &apos;-&apos; in them to have &apos;_&apos; instead due to how
    python unittest loader sees (or does not) the files.
 * added modules using SetUpClass() to call their parent&apos;s SetUpClass()
 * changed self.log to self.logger

(the sed part is refkit specific but I noticed some parts of the transition is easy to automate/script. it worked with &apos;find meta-refkit* -wholename &apos;*selftest/cases/*.py&apos;|sed ...&apos; just OK)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>74655</commentid>
    <comment_count>8</comment_count>
    <who name="Libertad">libertad.gonzalez.de.la.cruz</who>
    <bug_when>2017-06-29 17:33:28 +0000</bug_when>
    <thetext>As I understand this in this bug the idea is to better describe the Basic architecture information of the oeqa test. On this wiki https://wiki.yoctoproject.org/wiki/Oe-selftest. Describe how a new person can contribute to creating testcases for their new modules.

Make documentation proposal on this framework that later we can decide to add it to the mega manual or some other place. We will decide upon the content. 

The core development of this framework under 
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/core

Start by reading the README then the context.py

http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85902</commentid>
    <comment_count>9</comment_count>
    <who name="Armin Kuster">akuster</who>
    <bug_when>2019-12-22 18:37:10 +0000</bug_when>
    <thetext>we should still do this</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>87528</commentid>
    <comment_count>10</comment_count>
    <who name="Mark Morton">mark.morton</who>
    <bug_when>2020-06-19 13:07:11 +0000</bug_when>
    <thetext>I completed the first version of the new test-manual source and added it to the yocto-docs repo. There are a few minor updates, but the document itself is complete.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>