Bug 13165

Summary: Ptest: add binary PASS/FAIL scheme for use during build.
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Armin Kuster <akuster>
Component: coreAssignee: Armin Kuster <akuster>
Status: CLOSED NOTABUG QA Contact:
Severity: enhancement    
Priority: Medium CC: meta.mr.watcher, meta.watcher
Version: 5.99   
Target Milestone: 2.7   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Armin Kuster 2019-02-01 22:25:23 UTC
To help trigger changes in the AB, maybe the concept of expected pass/fail can be implemented at the recipe level.

example:

If we know a recipe that has ptest support has passed in the past, we can set a flag "PTEST_expected = "PASS" and if that changes during a AB ptest step, we fail the build so swat and friends can act upon it.

if we know a recipe has failures maybe "PTEST_expected_{machine} = "FAIL" or something crazy the includes a # 4 out of 22 test should fail.

just a thought...
Comment 1 Armin Kuster 2019-02-07 16:13:56 UTC
I could not find a bug for addressing this idea. Bug 12654 is for the overall QA process not just ptest.
Comment 2 Armin Kuster 2019-02-07 16:15:37 UTC
re-purposing this for comparing one ptest run against another to trigger PASS/FAIL in the AB.
Comment 3 Armin Kuster 2019-02-11 16:58:24 UTC
seems like a dup.
Comment 4 Armin Kuster 2019-02-11 16:58:32 UTC
.