Current doc [0], only mention output format: > The test generates output in the format used by Automake: > result: testname > where the result can be PASS, FAIL, or SKIP, and the testname can be any identifying string. ... but ptest-runner2 code uses exit code to determine if a test failed or not[1]: waitpid(child, &status, 0); if (WIFEXITED(status)) { exit_code = WEXITSTATUS(status); if (exit_code) { fprintf(fp, "\nERROR: Exit status is %d\n", exit_code); rc += 1; } Maybe we need to add "run-ptest must exit with status code 0 only if all test cases pass. If one or more tests fail, it must return a non-zero exit code." ? [0]: https://docs.yoctoproject.org/dev/test-manual/ptest.html#testing-packages-with-ptest [1]: https://git.yoctoproject.org/ptest-runner2/tree/utils.c#n529
While ptest-runner itself does not care about test output, testimage does. See * poky/meta/lib/oeqa/runtime/cases/ptest.py * poky/meta/lib/oeqa/utils/logparser.py having at least a "PASS:", "FAIL:", ... is expected. Without it, testimage gives an error: |AssertionError: |ptests which had no test results: |['rpm-sequoia'] See: https://lists.openembedded.org/g/openembedded-core/message/216032
Related email from Erik Schumacher: [ptest-runner][RESEND] Documentation and behavior do not match https://lists.yoctoproject.org/g/yocto-patches/message/1619 Le ven. 6 juin 2025 à 07:52, Erik Schumacher via lists.yoctoproject.org <erik.schumacher=iris-sensing.com@lists.yoctoproject.org> a écrit : I have created some ptests and executed them with the ptest-runner and realized that the documentation and the actual behavior of the ptest- runner don't really fit together. What I did: I used the wiki pages [1] and [2] as guidelines and also had a look at some existing run-ptest scripts like [3] and [4]. It's not entirely clear in the phrasing, but the documentation implies that a run-ptest script must print either "PASS", "FAIL" or "SKIP" followed by the testname. That's what most of the existing run-ptest scripts do. Then I created my own run-ptest script that did something like: /path/to/mybinary-gtest && echo "PASS: mybinary" || echo "FAIL: mybinary" What I expected: The script itself worked and ptest-runner was able to run it. So far, so good. I started to wonder when my script simulated a faulty test. My script printed "FAIL: mybinary", but ptest-runner completed the test without failures (last line: "TOTAL: 1 FAIL: 0"). I expected "FAIL: 1" here. The same applies to the xml output, which can be consumed by Gitlab when the ptest-runner runs inside a CI/CD pipeline. What I found: By looking at the code of ptest-runner, it quickly became clear to me that a failure is only determined by the exit code of a run-ptest script. See [5]. It doesn't matter what the script is printing. As long as the exit code is 0, the test has passed. Any other exit code is treated as fail. Why I think this is a problem: The behavior of ptest-runner somehow contradicts the documentation, which describes what the output of the script should look like. The documentation says nothing about the exit code being relevant. There is no run-ptest script that explicitly returns the right exit code. There are some that implicitly pass the exit code of the only / or last command, like gzip [6]. Using the generated xml file in a Gitlab testing pipeline will lead to a false sense of security that all tests would have been successful, even if they might not be. By relying on the exit code, ptest-runner also treats a run-ptest script as a single unit which can either pass or fail, even though a ptest might consists of multiple test cases ([3] or [4]). [1] https://wiki.yoctoproject.org/wiki/Ptest [2] https://docs.yoctoproject.org/test-manual/ptest.html [3] https://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/bc/bc/run-ptest [4] https://git.openembedded.org/openembedded-core/tree/meta/recipes-connectivity/bluez5/bluez5/run-ptest [5] https://git.yoctoproject.org/ptest-runner2/tree/utils.c#n522 [6] https://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gzip/files/run-ptest
A patch referring to this bug was merged in the docs: https://git.yoctoproject.org/yocto-docs/commit/?id=0d1fd79019883f366d796b58a01679297d7a5508 Yoann, do you consider this enough to close this ticket, or should we reiterate? I think the patch is an improvement, but is it precise enough in your opinion?
(In reply to Antonin Godard from comment #3) > A patch referring to this bug was merged in the docs: > https://git.yoctoproject.org/yocto-docs/commit/ > ?id=0d1fd79019883f366d796b58a01679297d7a5508 > > Yoann, do you consider this enough to close this ticket, or should we > reiterate? I think the patch is an improvement, but is it precise enough in > your opinion? Definitely a step in the right direction, but not enough in my humble opinion. What I would love is a kind of specification of the interface that must be implemented by the run-ptest script. Something like: > * return 0 if every sub-test passes > * return != 0 if a sub-test fails > * print at least a PASS:/FAIL:/SKIP: line (ideally one for every sub-test)
> Definitely a step in the right direction, but not enough in my humble opinion. I second this. Changing the documentation does not fix the existing differences between the output of run-ptest scripts and the expectations of the ptest-runner. Since the run-ptest scripts all work according to the same scheme, the ptest-runner should be adapted, in my opinion. The effort involved in changing all scripts, together with the risk of introducing new bugs, is significantly greater than adapting the ptest-runner. Regardless of this, the ptest-runner also has other problems. I started a discussion on the mailing list: https://lists.yoctoproject.org/g/yocto-patches/topic/ptest_runner_rfc_implement/113964431
(In reply to Erik Schumacher from comment #5) > > Definitely a step in the right direction, but not enough in my humble opinion. > > I second this. > > Changing the documentation does not fix the existing differences between the > output of run-ptest scripts and the expectations of the ptest-runner. Since > the run-ptest scripts all work according to the same scheme, the > ptest-runner should be adapted, in my opinion. The effort involved in > changing all scripts, together with the risk of introducing new bugs, is > significantly greater than adapting the ptest-runner. Regardless of this, > the ptest-runner also has other problems. > > I started a discussion on the mailing list: > https://lists.yoctoproject.org/g/yocto-patches/topic/ > ptest_runner_rfc_implement/113964431 Yeah as commented previously on the ML [1], I'm in favor to add an option (--strict?) to execute ptest-runner with log analyzer on output at post-process stage. Having it as an option enables us to look at specific ptest issues. [1] https://lists.yoctoproject.org/g/yocto-patches/message/1622
Hi, Right now the documentation says: > During the execution ptest-runner keeps count of total and failed ptests. At end the execution summary is written to the console. If any of the run-ptest fails, ptest-runner returns ‘1’. I've verified and this is still the current behavior. So, from a documentation standpoint, I think we're ok and we can close this bug. As stated in the previous comments, if there is a need to improve ptest-runner, please create another bug for it with the proper category, so the right person can try and work on it, or please adapt this one and let me know (so I don't close it). I will close this one week from now if there are no other comments. Thanks, Antonin
(In reply to Antonin Godard from comment #7) > Hi, > > Right now the documentation says: > > > During the execution ptest-runner keeps count of total and failed ptests. At end the execution summary is written to the console. If any of the run-ptest fails, ptest-runner returns ‘1’. > > I've verified and this is still the current behavior. So, from a > documentation standpoint, I think we're ok and we can close this bug. > > As stated in the previous comments, if there is a need to improve > ptest-runner, please create another bug for it with the proper category, so > the right person can try and work on it, or please adapt this one and let me > know (so I don't close it). > > I will close this one week from now if there are no other comments. IMHO, the current doc still lacks a crucial point : run-ptest must print at least a "PASS:", "FAIL:", ... (either by itself, or via a subprocess). The 0/!=0 return code is for ptest-runner, the "PASS:" lines are for testimage/logparser.
(In reply to Yoann Congal from comment #8) > IMHO, the current doc still lacks a crucial point : run-ptest must print at > least a "PASS:", "FAIL:", ... (either by itself, or via a subprocess). > The 0/!=0 return code is for ptest-runner, the "PASS:" lines are for > testimage/logparser. I sent a patch to mention this and clearly state these requirements: https://lore.kernel.org/r/20260109-ptest-output-v1-1-29b1b2039c97@bootlin.com
Patch merged, closing https://git.yoctoproject.org/yocto-docs/commit/?id=9292f61d7ba89598c89033ea7ee3b11a20d873f3