Bug 13165 - Ptest: add binary PASS/FAIL scheme for use during build.
Summary: Ptest: add binary PASS/FAIL scheme for use during build.
Status: CLOSED NOTABUG
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 5.99
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: 2.7
Assignee: Armin Kuster
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-02-01 22:25 UTC by Armin Kuster
Modified: 2019-02-11 16:58 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Kuster 2019-02-01 22:25:23 UTC
To help trigger changes in the AB, maybe the concept of expected pass/fail can be implemented at the recipe level.

example:

If we know a recipe that has ptest support has passed in the past, we can set a flag "PTEST_expected = "PASS" and if that changes during a AB ptest step, we fail the build so swat and friends can act upon it.

if we know a recipe has failures maybe "PTEST_expected_{machine} = "FAIL" or something crazy the includes a # 4 out of 22 test should fail.

just a thought...
Comment 1 Armin Kuster 2019-02-07 16:13:56 UTC
I could not find a bug for addressing this idea. Bug 12654 is for the overall QA process not just ptest.
Comment 2 Armin Kuster 2019-02-07 16:15:37 UTC
re-purposing this for comparing one ptest run against another to trigger PASS/FAIL in the AB.
Comment 3 Armin Kuster 2019-02-11 16:58:24 UTC
seems like a dup.
Comment 4 Armin Kuster 2019-02-11 16:58:32 UTC
.