Hello,

I am thinking that in order for this feature to be implementable you will most probably need some sort of power management. If the device is hanged or frozen a sure way to reboot it would be to do a power cycle. 
One ideea that comes to my mind is to modify the timeout decorator so that if the timeout is reached, then it will reset the board and try to continue after reboot. That should work fine for qemu but for actual boards to be able to reset them you will have to override the restart() method to use your own power cycle system.
To avoid the need to decorate every testcase with the timeout decorator I could decorate the whole oeTest class so it will automatically be applied to all TCs by default.

Do you think that this would be a viable option for you?

I agree timeout is the right place to trigger failsave approach. However, just decorate the whole oeTest class is very coarse granularity. I think each Test method should have a default implicit timeout which is injected by testrunner. If TC has special timeout, just mark it with a explicit decorator.

POC: http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lucian/oetest-error-handle

In this case just ignore the last commit (decorated oeTest class) and override the restart method in targetcontrol with whatever power management you have. The testcases would be deocorated with @timeout_reset(<seconds>).

Hi Musat, thanks for adding timeout from case level, that will be very useful I believe. When run into timeout, we might not reset the device always, can we change the decorate name from timeout_reset() to something like timeout_handler()?

Hi Cathy,
Yes, of course you can name it how ever you want.

If I understand right, when timeout happen, it trigger target.restart(). However, in real error handling, restart is the last thing we had to resort to. Thus, I'd suggest add a dedicate interface function in TestController for error handling , named sth like .failsafe(). When timeout happen, it trigger failsave() instead of restart().

Hello,

I created a working proof of concept here:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=lucian/timeout_handler&id=09361c1508cd1ca0add79d96f7f99fe3bf9c484a

I created a simple test suite, with 4 tests in which one has a sleep longer that the timeout set in the decorator in order to simulate a freeze. 
I rewrote the self.target.restart method with a method that in my case is using our power cycle script from our hardware lab.

By the third test the timeout runs out and then the NUC that I am testing on is rebooted and after that it continues with the fourth test.

I chose the self.target.restart in order to be compliant with the qemu interface (which was already working) but I guess a new method could be created in both classes (QemuTarget and RemoteTarget) in order to not use target.restart().

Testimage output:

| NOTE: Test modules  ['oeqa.runtime.poc']
| NOTE: Found 4 tests
| test_sleep_1min (oeqa.runtime.poc.PocTest) ... ok
| test_sleep_2min (oeqa.runtime.poc.PocTest) ... ok
| test_sleep_3min (oeqa.runtime.poc.PocTest) ... 
| ERROR
| test_sleep_4min (oeqa.runtime.poc.PocTest) ... ok
| 
| ======================================================================
| ERROR: test_sleep_3min (oeqa.runtime.poc.PocTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/home/lucian/work/SSD/work/poky/meta/lib/oeqa/utils/decorators.py", line 240, in wrapped_f
|     return fn(self, *args, **kw)
|   File "/home/lucian/work/SSD/work/poky/meta/lib/oeqa/runtime/poc.py", line 40, in test_sleep_3min
|   File "/home/lucian/work/SSD/work/poky/meta/lib/oeqa/utils/decorators.py", line 236, in raiseTimeOut
|     raise TimeOut('%s seconds' % seconds)
| TimeOut: 120 seconds
| 
| ----------------------------------------------------------------------
| Ran 4 tests in 544.088s
| 
| FAILED (errors=1)
| DEBUG: Python function do_testimage finished
| ERROR: Function failed: core-image-sato-sdk - FAILED - check the task log and the ssh log

If target.restart is currently workable for both qemu and real HW, I am fine with this solution right now. Later on, we may add a dedidate interface for error handling.

Patch was merged in commit d8eb9d41b0afb48f98dc59383cf886c8ca53c21a.

Close this one based on above commments.