| Summary: | Ptest-runner hangs on certain step | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [QA/Testing] Runtime Testing | Reporter: | Lucian Musat <georgex.l.musat> | ||||||
| Component: | general | Assignee: | Aníbal Limón <anibal.limon> | ||||||
| Status: | VERIFIED FIXED | QA Contact: | Cristina Agurida <cristina-danielax.agurida> | ||||||
| Severity: | major | ||||||||
| Priority: | Medium+ | CC: | alexandru.c.georgescu, benjamin.esquivel, bjst, bluelightning, cristian.iorga, francisco.j.pedraza.gonzalez, markus.boos, richard.purdie, ross.burton, tudor.florea | ||||||
| Version: | unspecified | ||||||||
| Target Milestone: | 2.1 M3 | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | |||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||
| Attachments: |
|
||||||||
|
Description
Lucian Musat
2015-07-16 11:41:40 UTC
Looks like lttng failing. Has anyone verified this by just running those tests? Hi Ross, Indeed the test that it's hanging is lttng-tools. It hangs at fast_regression testrun, testcase 277 or 278. The log looks like this when timeout kicks in and kills the process: PASS: 277 - Validate trace filter output PASS: 278 - Create sesEND: /usr/lib/lttng-tools/ptest Makefile:671: recipe for target 'installcheck-am' failed Makefile:405: recipe for target 'installcheck-recursive' failed A patch for this same issue was commited about a year ago: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=39f77bb6aaf4fab6eff2b32470800e183d2b7eaf Moving to M3 after discussion with Richard. Lowering priority since we have identified the test with the issue and can work around it during testing. Investigation started. Created attachment 2884 [details]
dmesg core-image-minimal
Hi Lucian,
I started to look into this and after run ptest-runner on qemux86 core-image-minimal a see the ooo-killer killing a python process, is this the issue?
alimon
Anibal, is that the same as the out of memory killer? (In reply to comment #8) > Anibal, is that the same as the out of memory killer? Yes, see the attached dmesg log. Talking with Lucian...
We consider that the main problem of ptest-runner for get blocked, is because the script iterates over the upstream package tests and runs it, if some upstream package test gets blocked, the ptest-runner blocks indefinitely.
So i'm going to re-implement ptest-runner (will be 2.0) in python with some enhancements,
- Use pool of workers for speed-up the execution.
- Add a timeout policy this will be based on state of the process and stdout.
Some thing to consider is the dependency of the new ptest-runner on python.
Comments?
Cheers,
alimon
The only issue with rewriting it in python is that python then will be required on the target to run tests, which seems like quite a bit of overhead for something like this. Happy to hear other opinions on this though. For what it's worth coreutils and busybox have a "timeout" command to kill a process after N seconds. Maybe it's worth doing both: adding use of timeout to the shell implementation so it can't hang *forever*, and write a Python runner for systems where Python isn't too large? What about using native binaries where python is not present? I've heard of py2exe and pyinstaller for embedding the python interpreter with the code into a binary executable (In reply to comment #12) > For what it's worth coreutils and busybox have a "timeout" command to kill a > process after N seconds. Maybe it's worth doing both: adding use of timeout > to the shell implementation so it can't hang *forever*, and write a Python > runner for systems where Python isn't too large? The problem with the timeout command is that kills the process after run certain seconds and we don't know how many time the upstream package test suite needs to run. I addressed this issue using a timeout over stdout of the test [1], so if don't report any data in certain seconds the test is treaty as a hang and ptest-runner kills it. [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py?h=alimon/ptest-runner&id=bc230a8043e05c0a6673b7591359b446bae91adf#n95 The problem with calculating the 'typical' time over console output (stdout/stderr) is no different than the problem of calculating the same for the entire test. The proposed solution which I understand as assuming that if the test is not putting any output to the console therefore is a hung test is very likely to prove itself faulty soon or very soon. A test could have no output at all for X time and still be able to finish by time Y (where Y > X) and that doesn't necessarily mean it was hung. One of the few testing conventions that have proven useful is to always test the output of the command ($?) instead of stdout/stderr analysis The test timeout strategy is never to have a fixed number for all tests but rather be flexible about it, you might have a default timeout which works for most of your tests but you need to be able to tag individually tests in case they take longer and it is justified (In reply to comment #15) > The problem with calculating the 'typical' time over console output > (stdout/stderr) is no different than the problem of calculating the same for > the entire test. The proposed solution which I understand as assuming that > if the test is not putting any output to the console therefore is a hung > test is very likely to prove itself faulty soon or very soon. A test could > have no output at all for X time and still be able to finish by time Y > (where Y > X) and that doesn't necessarily mean it was hung. One of the few > testing conventions that have proven useful is to always test the output of > the command ($?) instead of stdout/stderr analysis I also use the $? value for store if program execution was successful but for this case i think is different to have a timeout for all the execution and for stdout since upstream tests are indented to show results on stdout so we can have a spaces of maximum 5 minutes (this is the default) for every byte to stdout in other way we need to define a bigger timeout for the whole the process. It's a bit complicated but i think timeouted the stdout is better than the whole process, the stdout timeout gives our more control. Another possible solution but it requires much more developer time is to modify the tests to provide a way to notify ptest-runner that the test is running and not hang. > The test timeout strategy is never to have a fixed number for all tests but > rather be flexible about it, you might have a default timeout which works > for most of your tests but you need to be able to tag individually tests in > case they take longer and it is justified Agree. Finally i'll add the recipe of ptest-runner in python and set as default the shell one. Great, this is an interesting change, I'll keep an eye on it Sent v3 with few change in recipe install also with default install 1.0 in poky-tiny distro. http://lists.openembedded.org/pipermail/openembedded-core/2016-January/114772.html Due to a discussion in the ML [1] about ptest-runner in python and tiny system. I decided to write a version in C with all the features supported for the python version [2], i expect to finish it this week for review. The repo [2] for now is on github when finish i'll request a repo in git.yp.org. [1] http://lists.openembedded.org/pipermail/openembedded-core/2016-January/114913.html [2] https://github.com/alimon/ptest-runner Merge at, http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=209066ccce1f211b7982372dcb9b9795493aaece Cristina, Francisco, can you verify this? Verified with yocto-2.1_M2.rc1, poky: 23056103c949b498c23b47579e8dd57ce78e6ed9 |