Ross has fixed a number of glib ptest so he may provide advice.

Now seen on the arm host too.

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1716

was able to reproduce 
standard build of core-image-sato-sdk
although it has flaky/intermittent behaviour
Run 3 times, fails once
```
ERROR: Exit status is 512
DURATION: 3
END: /usr/lib/glib-2.0/ptest
2021-01-29T15:32
STOP: ptest-runner
```

was able to narrow it down to
__main__.TestCodegen.test_reproducible failing

I think this is an issue with the amount of output generated by this particular test.


if i pipe the output of the ptest-runner into a file
`ptest-runner > test.log`, the test passes

however,
if i do `ptest-runner`, the test fails saying exit status 512.

i think this is related to `TestCodegen.test_reproducible` generating around 6000 lines of just boiler plate code.

root cause is in `ptest-runner`. it seems to have trouble dealing with large amount of data coming from the child process.

this does not happen if you do `ptest-runner > filename`

to reproduce this type of issue:

use `ptest-runner` to run the following script

echo -e {1..100000}\\n 


ptrace shows this before failing (child process probably crashed)
```
read(3, 0x7ffc8e9a08c0, 1024)           = -1 EAGAIN (Resource temporarily unavailable)
wait4(21170, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 21170
poll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}], 2, 200) = 0 (Timeout)
write(1, "\n", 1
)                       = 1                                    
write(1, "ERROR: Exit status is 256\n", 26ERROR: Exit status is 256

```

2 proposed fix for this:

1. remove unecessary print statements from glib


2. change `ptest-runner` to do blocking IO on pipes. this will make sure the child process stalls if the pipe is full.

option 2 of fixing ptest-runner would be better so that we don't have to find and tweak other programs that behave the same way.

ok the root_root cause of the issue is a 

*read from a non-blocking pipe, but
*write to a blocking stdout

so when the data becomes too large, it would block the write and the pipe will be full.

The child process writing to the non-blocking pipe will sometimes crash because it gets an error EAGAIN. It doesn't handle it well because it thinks it is writing to blocking stdout

```
/home/yyu1/ptest-runner2/lol/ptest/run-ptest: line 10: echo: write error: Resource temporarily unavailable
```

although not mentioned in the patch, i did submit the change to upstream
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1929

glib rejected the change. see previous link for a response.

one interesting suggestion is to have testrunner ignore certain outputs since all output glib creates follow the tap convention of commenting out these debugging outputs with '#'

so run-ptest for glibc could specifically ignore with grep -v '^#'

Note that a work-around for this bug is already in oe-core master:

http://cgit.openembedded.org/openembedded-core/commit/?id=afc9ba7d546f3f2e60fb6f46f740dc925542df16

We're waiting on a fix from Anibal for ptest-runner to revert the work-around.

(In reply to comment #13)
> Note that a work-around for this bug is already in oe-core master:
> 
> http://cgit.openembedded.org/openembedded-core/commit/
> ?id=afc9ba7d546f3f2e60fb6f46f740dc925542df16
> 
> We're waiting on a fix from Anibal for ptest-runner to revert the
> work-around.

I have the patches [1] and tested  with glib-2.0 ptest but fails after 958 secs on qemuarm64 in glib-threading tests, that will require to implement per test timeouts because only global timeout is supported default to 300 secs can be override by -t.

The idea is to add a special file under ptest folder like for example in glib-2.0,

/usr/lib/glib-2.0/ptest/timeout

If that file exists in the ptest use the timeout from there if not use global one.


[1] https://github.com/alimon/ptest-runner2/tree/ptest_fixes

(In reply to comment #14)
> I have the patches [1] and tested  with glib-2.0 ptest but fails after 958
> secs on qemuarm64 in glib-threading tests, that will require to implement
> per test timeouts because only global timeout is supported default to 300
> secs can be override by -t.

I think if we do the "reset timeout timer as long as there is output", we are fine for glib-2.0

preparing a patch to revert the workaround.

https://git.openembedded.org/openembedded-core/commit/?id=e7be3901e43645796e195348924739d03495a079

reverted the workaround.

https://git.openembedded.org/openembedded-core/commit/?id=e7be3901e43645796e195348924739d03495a079

reverted the workaround.