Bug 13594

Summary: [3.0 RC2] mpc8315e-rdb: the stap oeqa test causes OOM
Product: [Build System, Metadata & Runtime] BSPs Reporter: Yi Zhao <yi.zhao>
Component: bsps-meta-yoctoAssignee: Kevin Hao <kexin.hao>
Status: RESOLVED OBSOLETE QA Contact: Yi Zhao <yi.zhao>
Severity: normal    
Priority: Medium CC: ee.peng.yeoh, randy.macleod, richard.purdie, sangeeta.jain
Version: 3.0   
Target Milestone: 3.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: Regression (Used to work)
Verified: Documentation change: No (bug/feature does not impact docs)

Description Yi Zhao 2019-10-15 01:30:45 UTC
Git rev: master/94f6b31befda5c496f65e863a6f8152b42d7ebf0
Image location: https://autobuilder.yocto.io/pub/releases/yocto-3.0.rc2/machines/mpc8315e-rdb/

The stap oeqa test case is failed:

"stap.StapTest.test_stap": {

"log": "Traceback (most recent call last):\n  File \"/buildarea/poky/meta/lib/oeqa/core/decorator/__init__.py\", line 36, in wrapped_f\n    return func(*args, **kwargs)\n  File \"/buildarea/poky/meta/lib/oeqa/core/decorator/__init__.py\", line 36, in wrapped_f\n    return func(*args, **kwargs)\n  File \"/buildarea/poky/meta/lib/oeqa/core/decorator/__init__.py\", line 36, in wrapped_f\n    return func(*args, **kwargs)\n  File \"/buildarea/poky/meta/lib/oeqa/runtime/cases/stap.py\", line 37, in test_stap\n    self.assertEqual(status, 0, msg='\\n'.join([cmd, output]))\nAssertionError: 137 != 0 : stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp\nWARNING: Kernel function symbol table missing [man warning::symbols]\nsh: line 1: 19933 Killed                  stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp\n",

"status": "FAILED"
}

There are some OOM messages showed on target:
#######################################################
hellomod: loading out-of-tree module taints kernel.
Hello world!
Cleaning up hellomod.
cc1 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 0 PID: 24633 Comm: cc1 Tainted: G           O      5.2.17-yocto-standard #1
Call Trace:
[c5c23c58] [c01b207c] dump_header+0x4c/0x2c4 (unreliable)
[c5c23ca8] [c01b127c] oom_kill_process+0x234/0x238
[c5c23cc8] [c01b1e18] out_of_memory+0x268/0x3bc
[c5c23cf8] [c020b8e0] __alloc_pages_nodemask+0xbe4/0xe48
[c5c23dc8] [c01a8ec0] pagecache_get_page+0x164/0x3b4
[c5c23e08] [c01aab44] filemap_fault+0x38c/0x8dc
[c5c23e48] [c01eb858] __do_fault+0x58/0xf0
[c5c23e68] [c01f1b70] handle_mm_fault+0xb18/0x1014
[c5c23ee8] [c001cfc4] __do_page_fault+0x168/0xbb8
[c5c23f38] [c00163ac] handle_page_fault+0x14/0x40
--- interrupt: 401 at 0x10209f24
    LR = 0x10378148
Mem-Info:
active_anon:21918 inactive_anon:48 isolated_anon:0
 active_file:70 inactive_file:389 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:1497 slab_unreclaimable:1211
 mapped:365 shmem:256 pagetables:206 bounce:0
 free:809 free_pcp:34 free_cma:0
Node 0 active_anon:87672kB inactive_anon:192kB active_file:280kB inactive_file:1556kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1460kB dirty:0kB writeback:0kB shmem:1024kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Normal free:3236kB min:1348kB low:1684kB high:2020kB active_anon:87672kB inactive_anon:192kB active_file:280kB inactive_file:1556kB unevictable:0kB writepending:0kB present:131072kB managed:114340kB mlocked:0kB kernel_stack:744kB pagetables:824kB bounce:0kB free_pcp:136kB local_pcp:136kB free_cma:0kB
lowmem_reserve[]: 0 0
Normal: 372*4kB (UM) 109*8kB (UM) 27*16kB (UM) 13*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3208kB
722 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
32768 pages RAM
0 pages HighMem/MovableOnly
4183 pages reserved
Tasks state (memory values in pages):
[  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[    145]     0   145     1043      148    28672        0         -1000 udevd
[    664]   999   664      956       63    28672        0             0 dbus-daemon
[    668]     0   668     3375       54    24576        0             0 rngd
[    685]     0   685     2996      130    36864        0             0 connmand
[    691]     0   691     2874       80    40960        0             0 wpa_supplicant
[    698]     0   698     1822       87    20480        0         -1000 sshd
[    712]   998   712      785       40    16384        0             0 rpcbind
[    721]     0   721      623       29    20480        0             0 atd
[    742]   997   742     1194       82    28672        0             0 avahi-daemon
[    743]   997   743     1161       49    24576        0             0 avahi-daemon
[    755]     0   755     2086      100    28672        0             0 ofonod
[    761]     0   761      727       64    20480        0             0 crond
[    772]     0   772    12659      286    49152        0             0 tcf-agent
[    779]     0   779     1012       46    24576        0             0 start_getty
[    780]     0   780     1148       32    24576        0             0 getty
[    794]     0   794     1225      233    24576        0             0 sh
[  23037]     0 23037      988      200    20480        0             0 syslogd
[  23040]     0 23040      989      207    20480        0             0 klogd
[  24013]     0 24013     1822       97    20480        0             0 sshd
[  24015]     0 24015     1011       41    24576        0             0 sh
[  24016]     0 24016    13385    10216    81920        0             0 stap
[  24017]     0 24017      745      113    20480        0             0 make
[  24320]     0 24320      805      174    20480        0             0 make
[  24629]     0 24629     1013       64    24576        0             0 sh
[  24631]     0 24631      912       48    20480        0             0 gcc
[  24633]     0 24633    16852     9223    94208        0             0 cc1
[  24634]     0 24634     1827      625    32768        0             0 as
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=stap,pid=24016,uid=0
Out of memory: Killed process 24016 (stap) total-vm:53540kB, anon-rss:40840kB, file-rss:24kB, shmem-rss:0kB
oom_reaper: reaped process 24016 (stap), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

##############################################################


But when I run this case manually on the target, the case can pass.

Steps:

$ cat hello.stp 
probe oneshot { println("hello world") }

$ cd /usr/src/kernel && make scripts prepare
$ cd /lib/modules/`uname -r` && (if [ ! -e build ]; then ln -s /usr/src/kernel build; fi)
$ stap --disable-cache -DSTP_NO_VERREL_CHECK /tmp/hello.stp
Comment 1 Randy MacLeod 2019-10-17 14:37:27 UTC
Kevin, Can you see if disabling tcf-agent and other non-essential services avoids the OOM?
Seems like the hardware just needs more RAM.
Comment 2 Richard Purdie 2019-10-17 14:37:51 UTC
We could try disabling tcf-agent and/or wpa_supplicant on this platform since we probably don't need these during the test.
Comment 3 Kevin Hao 2020-04-13 23:30:23 UTC
This board has been dropped from the meta-yocto-bsp