Bug 13556 - [2.8 M3 RC1] strace failure in ptest
Summary: [2.8 M3 RC1] strace failure in ptest
Status: RESOLVED FIXED
Alias: None
Product: Package Testing (ptest)
Classification: QA/Testing
Component: ptest (show other bugs)
Version: unspecified
Hardware: x86 x86_64
: High normal
Target Milestone: 2.8 M4
Assignee: Bruce Ashfield
QA Contact:
URL:
Whiteboard:
: 13506 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-09-26 06:16 UTC by Sudhir Sharma
Modified: 2019-10-10 20:57 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments
regression_results (98.78 KB, text/plain)
2019-09-26 06:16 UTC, Sudhir Sharma
no flags Details
2.8_M3_RC1_Report (20.23 KB, text/plain)
2019-09-26 06:17 UTC, Sudhir Sharma
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sudhir Sharma 2019-09-26 06:16:29 UTC
Created attachment 4568 [details]
regression_results

We took ptest results from autobuilder and did ptest regression. 

1. Download ptest results for current and previous release.
2. Run resulttool regression for ptest
3. Identify test that passed previous release and failed  this release

Previous release 
https://autobuilder.yocto.io/pub/releases/yocto-2.8_M2.rc1/testresults/qemux86-64-ptest/

Current release
https://autobuilder.yocto.io/pub/releases/yocto-2.8_M3.rc1/testresults/qemux86-64-ptest/

We have observed failure in ptest regression results.

Pango
ptestresult.pango.pango/testiter.test: FAILED -> PASSED

Strace
ptestresult.strace.clock.gen.test: FAILED -> PASSED
ptestresult.strace.detach-stopped.test: FAILED -> PASSED
ptestresult.strace.ipc_msgbuf-Xraw.gen.test: FAILED -> PASSED
ptestresult.strace.ipc_shm-Xabbrev.gen.test: FAILED -> PASSED
ptestresult.strace.ipc_shm-Xverbose.gen.test: FAILED -> PASSED
ptestresult.strace.ipc_shm.gen.test: FAILED -> PASSED
ptestresult.strace.qual_fault-syscall.test: FAILED -> PASSED
ptestresult.strace.qual_fault.test: FAILED -> PASSED
ptestresult.strace.regex.gen.test: FAILED -> PASSED
ptestresult.strace.sched.gen.test: FAILED -> PASSED
ptestresult.strace.sigblock.test: FAILED -> PASSED
ptestresult.strace.sigign.test: FAILED -> PASSED
ptestresult.strace.trace_fstat.gen.test: FAILED -> PASSED
ptestresult.strace.trace_fstatfs.gen.test: FAILED -> PASSED
ptestresult.strace.trace_lstat.gen.test: FAILED -> PASSED
ptestresult.strace.trace_personality_32.gen.test: FAILED -> PASSED
ptestresult.strace.trace_personality_64.gen.test: FAILED -> PASSED
ptestresult.strace.trace_personality_regex_32.gen.test: FAILED -> PASSED
ptestresult.strace.trace_personality_regex_64.gen.test: FAILED -> PASSED
ptestresult.strace.trace_personality_regex_x32.gen.test: FAILED -> PASSED
ptestresult.strace.trace_personality_x32.gen.test: FAILED -> PASSED
ptestresult.strace.trace_question.gen.test: FAILED -> PASSED
ptestresult.strace.trace_stat.gen.test: FAILED -> PASSED
ptestresult.strace.trace_stat_like.gen.test: FAILED -> PASSED
ptestresult.strace.trace_statfs.gen.test: FAILED -> PASSED
ptestresult.strace.trace_statfs_like.gen.test: FAILED -> PASSED

Attaching failure log for pango and strace.
Comment 1 Sudhir Sharma 2019-09-26 06:17:07 UTC
Created attachment 4569 [details]
2.8_M3_RC1_Report
Comment 2 Armin Kuster 2019-09-26 15:12:11 UTC
make this strace only
Comment 3 Stephen K Jolley 2019-09-26 15:13:29 UTC
*** Bug 13506 has been marked as a duplicate of this bug. ***
Comment 4 Richard Purdie 2019-09-27 12:55:25 UTC
We do have a patch to upgrade strace to version 5.3, test run here:
https://autobuilder.yocto.io/pub/non-release/20190926-14/testresults/qemux86-64-ptest/strace.log
however the failures are still present.
Comment 5 Randy MacLeod 2019-10-03 01:42:55 UTC
There's a thread on lkml: 
   https://lkml.org/lkml/2019/10/1/1486

I was curious about what was going wrong in the kernel in this case so I tried using 'perf trace' as a first warm-up step since my perf-foo is not good.
I was surprised to see:

# time strace -o log -qq -esignal=none -e/clock ./printpath-umovestr > ttt
-- 4m11s
# time perf trace -o perf-stace-printpath.log strace -o under-perf-log -qq -esignal=none -e/clock ./printpath-umovestr > ttt-perf
--> 8 seconds

but

# cmp ttt ttt-perf
ttt ttt-perf differ: char 126658361, line 61426

the ttt-perf log line reads:
   chdir(0x7f6c80e99001) = -1 EFAULT (Bad address)

This happens when the length of the string passed to chdir() and maybe memmove() in:
test_printpath_at(char *const p, const unsigned int test_max)
...
		memmove(eop - i, eop - i_1, i_1);
		eop[-1] = "/.."[i_1 % 3];
		if (chdir(eop - i))
...
goes over the PATH_MAX limit of 4096.

That's likely where the slowdown starts but I haven't confirmed that yet.
I could change the test code to run 2 tests rather than 8192. I'd do N loops at length 4096-128 and at 4096+128 and time each or something simpler. I expect the times to be close on an older kernel and quite different on a newer buggy kernel. 

More tomorrow, albeit in between meetings. :-(
Comment 6 Bruce Ashfield 2019-10-08 15:25:22 UTC
Patch sent to list.
Comment 8 Hongxu Jia 2019-10-09 01:32:53 UTC
Here is the quick reproduce way

$ cat >> test.c << ENDOF
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[])
{
#define TEST_MAX 4096
    char p[TEST_MAX] = {0};
    char *const eop = p + (TEST_MAX - 1); 
    int i;
    for (i = 1; i < TEST_MAX; ++i) {
        const int i_1 = i - 1;
        memmove(eop - i, eop - i_1, i_1);
        eop[-1] = "/.."[i_1 % 3]; 
        printf("%s\n", eop - i); 
    }   

  return 0;
}
ENDOF

$ gcc test.c -o test

$ time strace  -e/clock ./test >ttt
real	0m8.386s
user	0m0.227s
sys	0m0.128s

$ time strace  -e/clock ./test >/dev/null
real	0m0.338s
user	0m0.033s
sys	0m0.043s
Comment 9 Hongxu Jia 2019-10-09 01:35:42 UTC
After applying the fix on kernel source
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -2155,8 +2155,8 @@ static void ptrace_stop(int exit_code, int why, int clear_code, kernel_siginfo_t
                 */
                preempt_disable();
                read_unlock(&tasklist_lock);
-               preempt_enable_no_resched();
                cgroup_enter_frozen();
+               preempt_enable_no_resched();
                freezable_schedule();
                cgroup_leave_frozen(true);
        } else {

Here is the test result:
root@qemux86-64:~# time strace  -e/clock ./test >ttt
+++ exited with 0 +++

real	0m0.132s
user	0m0.028s
sys	0m0.100s
root@qemux86-64:~# time strace  -e/clock ./test >/dev/null
+++ exited with 0 +++

real	0m0.074s
user	0m0.022s
sys	0m0.050s
Comment 10 Bruce Ashfield 2019-10-09 02:50:48 UTC
I'm not sure why the status of this was changed back to NEW, but Richard has the patches for this and they are in the M4 build. Moving it back to "in progress review".
Comment 11 Hongxu Jia 2019-10-09 02:53:22 UTC
(In reply to comment #10)
> I'm not sure why the status of this was changed back to NEW, but Richard has
> the patches for this and they are in the M4 build. Moving it back to "in
> progress review".

Sorry, I incorrectly click on the save button
Comment 12 Richard Purdie 2019-10-10 12:22:23 UTC
Thanks everyone for the work on this, a good one to fix!

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=cdb9ea2a8b74d6675bd1af31835995a8f983bbf3
Comment 13 Randy MacLeod 2019-10-10 20:57:02 UTC
Also note that we are intentionally leaving the timeout in run-ptest at 30 seconds.