13411 2019-06-19 16:12:29 +0000 ptest-perl.bbclass run-ptest is too greedy for SKIP 2026-08-28 07:17:12 +0000 1 1 10 QA/Testing Package Testing (ptest) ptest 3.1 x86 Multiple RESOLVED FIXED Medium normal 6.99 1 tim.orling unassigned mathieu.dubois-briand randy.macleod --- --- Don't know oldest_to_newest 84253 0 tim.orling 2019-06-19 16:12:29 +0000 The current implementation will take the output of any test case and mark the entire suite as SKIP when "skip" or "SKIP" is in the test output. Many packages have tests that test if something is properly skipped. If a line starts with "ok" it should most likely be a PASS (regardless of the presence of the string "skip" in the output). Properly parse TAP output for actual outcome. 84623 1 tim.orling 2019-08-08 15:33:47 +0000 From the TAP specification [1]: DIRECTIVES Directives are special notes that follow a # on the test line. Only two are currently defined: TODO and SKIP. Note that these two keywords are not case-sensitive. TODO tests If the directive starts with # TODO, the test is counted as a todo test, and the text after TODO is the explanation. not ok 13 # TODO bend space and time Note that if the TODO has an explanation it must be separated from TODO by a space. These tests represent a feature to be implemented or a bug to be fixed and act as something of an executable “things to do” list. They are not expected to succeed. Should a todo test point begin succeeding, the harness should report it as a bonus. This indicates that whatever you were supposed to do has been done and you should promote this to a normal test point. Skipping tests If the directive starts with # SKIP, the test is counted as having been skipped. If the whole test file succeeds, the count of skipped tests is included in the generated output. The harness should report the text after # SKIP\S*\s+ as a reason for skipping. ok 23 # skip Insufficient flogiston pressure. Similarly, one can include an explanation in a plan line, emitted if the test file is skipped completely: 1..0 # Skipped: WWW::Mechanize not installed [1] https://testanything.org/tap-specification.html 84624 2 tim.orling 2019-08-08 15:39:54 +0000 The regex should be checking for test suites that have been skipped, which would be of the format: 1..0 # Skipped: WWW::Mechanize not installed But not test cases that have been skipped (which would start with 'ok', and have '# skip|SKIP' directive), which would be of the format: ok 23 # skip Insufficient flogiston pressure. We may want to skip or ignore failed TODO directive test cases. This pollutes ptest results with false negatives that are really only helpful to the author of the perl module. In practice individual *.t files can be skipped in the recipe, but this is a manual process and certainly brittle. 85303 3 tim.orling 2019-10-10 15:02:26 +0000 I have not yet found a workable solution and 3.0 is in QA, so bump to 3.1. 86098 4 tim.orling 2020-01-23 15:56:11 +0000 Alex patched the ptest for libmodule-build-perl, which is based on the same code. Apply the same fix to the general script. 94236 5 tim.orling 2022-10-24 22:42:55 +0000 Bump. Should be able to get around to this. 98761 6 tim.orling 2024-04-18 14:55:37 +0000 I don't seem to be getting to this, so moving to 5.99 and that Unassigned developer that has a large back log. 106668 7 mathieu.dubois-briand 2026-08-28 07:16:49 +0000 Patch merged: https://git.openembedded.org/openembedded-core/commit/?id=3e5939faa8f522fbf43ce0f3c5ccb07c1e5a46c2 106669 8 mathieu.dubois-briand 2026-08-28 07:17:12 +0000 Using correct status.