The current implementation will take the output of any test case and mark the entire suite as SKIP when "skip" or "SKIP" is in the test output. Many packages have tests that test if something is properly skipped. If a line starts with "ok" it should most likely be a PASS (regardless of the presence of the string "skip" in the output). Properly parse TAP output for actual outcome.
From the TAP specification [1]: DIRECTIVES Directives are special notes that follow a # on the test line. Only two are currently defined: TODO and SKIP. Note that these two keywords are not case-sensitive. TODO tests If the directive starts with # TODO, the test is counted as a todo test, and the text after TODO is the explanation. not ok 13 # TODO bend space and time Note that if the TODO has an explanation it must be separated from TODO by a space. These tests represent a feature to be implemented or a bug to be fixed and act as something of an executable “things to do” list. They are not expected to succeed. Should a todo test point begin succeeding, the harness should report it as a bonus. This indicates that whatever you were supposed to do has been done and you should promote this to a normal test point. Skipping tests If the directive starts with # SKIP, the test is counted as having been skipped. If the whole test file succeeds, the count of skipped tests is included in the generated output. The harness should report the text after # SKIP\S*\s+ as a reason for skipping. ok 23 # skip Insufficient flogiston pressure. Similarly, one can include an explanation in a plan line, emitted if the test file is skipped completely: 1..0 # Skipped: WWW::Mechanize not installed [1] https://testanything.org/tap-specification.html
The regex should be checking for test suites that have been skipped, which would be of the format: 1..0 # Skipped: WWW::Mechanize not installed But not test cases that have been skipped (which would start with 'ok', and have '# skip|SKIP' directive), which would be of the format: ok 23 # skip Insufficient flogiston pressure. We may want to skip or ignore failed TODO directive test cases. This pollutes ptest results with false negatives that are really only helpful to the author of the perl module. In practice individual *.t files can be skipped in the recipe, but this is a manual process and certainly brittle.
I have not yet found a workable solution and 3.0 is in QA, so bump to 3.1.
Alex patched the ptest for libmodule-build-perl, which is based on the same code. Apply the same fix to the general script.
Bump. Should be able to get around to this.
I don't seem to be getting to this, so moving to 5.99 and that Unassigned developer that has a large back log.
Patch merged: https://git.openembedded.org/openembedded-core/commit/?id=3e5939faa8f522fbf43ce0f3c5ccb07c1e5a46c2
Using correct status.