Bug 11368

Summary: Autobuilder ran out of memory during nightly-oe-selftest
Product: [Infrastructure] AutoBuilder Reporter: California Sullivan <california.l.sullivan>
Component: autobuilderAssignee: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Status: RESOLVED DUPLICATE QA Contact:
Severity: normal    
Priority: Medium CC: infras.ab.watcher, Infras.watcher, joshuagloe, monserratx.sedeno.bustos, pidge, ross.burton
Version: unspecified   
Target Milestone: 2.5   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Running oe-selftest log
none
CreateAutoConf log none

Description California Sullivan 2017-04-14 23:03:15 UTC 
Created attachment 3723 [details]
Running oe-selftest log

During nightly-oe-selftest, the autobuilder seemed to have ran out of memory, resulting in errors such as the following:

| gpg: signing failed: Cannot allocate memory
| gpg: signing failed: Cannot allocate memory
| error: gpg exec failed (2)

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/263/steps/Running%20oe-selftest/logs/stdio

Thinking it was a one-off issue, we are rerunning nightly-oe-selftest, but the system is still acting like its out of memory, with the log file currently hung with the following, and the server being very slow to serve the page:

2017-04-14 22:33:59 - test_continue (oeqa.selftest.bbtests.BitbakeTests) ... ok
2017-04-14 22:34:14 - test_dry_run (oeqa.selftest.bbtests.BitbakeTests) ... ok
2017-04-14 22:34:18 - test_environment (oeqa

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/264/steps/Running%20oe-selftest/logs/stdio
Comment 2 Ross Burton 2017-04-20 14:51:59 UTC 
*** Bug 11376 has been marked as a duplicate of this bug. ***
Comment 3 Ross Burton 2017-04-25 09:40:04 UTC 
debian-testing.yocto.io just failed a selftest run with out of memory errors:

gpg: signing failed: Cannot allocate memory

Both the original report and this instance were on debian-testing.
Comment 4 Michael Halstead 2017-04-25 17:41:50 UTC 
Memory stats from debian-testing during the failure.

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
12:05:01 AM  21213844 110713428     83.92  45293808  51996532   5712724      4.27  82778656  16433740  18776444
12:15:01 AM  16910928 115016344     87.18  46150612  55134784   4223720      3.15  84557068  19025368  15073672
12:25:01 AM  11213344 120713928     91.50  46913488  59537740   7175960      5.36  86122064  23135876  16337260
12:35:02 AM   3813744 128113528     97.11  47610936  66035756   7366764      5.50  91761116  24796784  20408688
12:45:01 AM   2414168 129513104     98.17  48314208  66713336  11028516      8.24  94394036  23663876  20274232
12:55:01 AM   1789368 130137904     98.64  48959436  67408864   4929552      3.68  96129108  22653472  20532012
01:05:01 AM   2016348 129910924     98.47  49862344  67152976   2405872      1.80  98057924  20580364  18614112
01:15:01 AM   3232520 128694752     97.55  50707428  63720396   7150840      5.34 103419816  13960936  15536576
01:25:01 AM   6338220 125589052     95.20  50960952  58982892  23157988     17.30 102354696  12059040   1873740
01:35:01 AM   1790680 130136592     98.64  50681956  62342052  18633972     13.92 109251944   9588000   1596932
01:45:01 AM   3750636 128176636     97.16  50793500  63678528   5228616      3.91 107416604   9263316    237664
01:55:01 AM   2617004 129310268     98.02  51104376  65289740   2026204      1.51 108368992   9283700      6960
02:05:01 AM   4915912 127011360     96.27  50912596  62661420   2375420      1.77 108380652   7001040    435796
02:15:01 AM   1757688 130169584     98.67  50277652  66671660   4922536      3.68 111347036   7170148    218844
02:25:01 AM   4578128 127349144     96.53  49474060  65530940   1159440      0.87 109395872   6373020      4136
02:35:01 AM   4062084 127865188     96.92  50015108  66164032    370092      0.28 110160692   6124280         0
02:45:01 AM   3366812 128560460     97.45  50740808  66131712    357460      0.27 110885928   6088784     20384
02:55:01 AM   3369912 128557360     97.45  51063848  65812540    344080      0.26 111168212   5810180         8
03:05:01 AM   3259000 128668272     97.53  51386160  65602924    377828      0.28 111446528   5644700         8
03:15:01 AM   3216560 128710712     97.56  51702112  65329040    384668      0.29 111607660   5524516         8
03:25:01 AM   3216652 128710620     97.56  51702124  65329060    351744      0.26 111608332   5524432         0
03:35:01 AM   3219248 128708024     97.56  51702136  65329084    342420      0.26 111607996   5524448         0
03:45:01 AM   3224388 128702884     97.56  51702148  65329104    319624      0.24 111608272   5524468         8
03:55:01 AM   3224284 128702988     97.56  51702152  65329128    298992      0.22 111608228   5524496         8
04:05:01 AM   3219736 128707536     97.56  51702172  65329148    405824      0.30 111608144   5524516         0
04:15:01 AM   3223116 128704156     97.56  51702200  65329172    316992      0.24 111607944   5524540         8
04:25:01 AM   3216844 128710428     97.56  51702208  65329192    299120      0.22 111612788   5524560         8
04:35:01 AM   3217664 128709608     97.56  51702992  65329220    365212      0.27 111609160   5524588         8
04:45:01 AM   3219232 128708040     97.56  51702996  65329240    369816      0.28 111609016   5524604         0
04:55:01 AM   3217636 128709636     97.56  51703016  65329260    374404      0.28 111609252   5524624         8
05:05:01 AM   3217468 128709804     97.56  51703024  65329284    321648      0.24 111608980   5524652         0
05:15:01 AM   3219132 128708140     97.56  51703036  65329308    370512      0.28 111609100   5524676         8
05:25:01 AM   3219828 128707444     97.56  51703052  65329332    362572      0.27 111609072   5524704         8
05:35:01 AM   3224084 128703188     97.56  51703088  65329352    353756      0.26 111608944   5524712         8
05:45:01 AM   3221440 128705832     97.56  51703096  65329372    305788      0.23 111609164   5524732         0
05:55:01 AM   3220404 128706868     97.56  51703100  65329392    397040      0.30 111609328   5524756        12
06:05:01 AM   3219000 128708272     97.56  51703132  65329412    377716      0.28 111609012   5524776         8
06:15:01 AM   3221064 128706208     97.56  51703140  65329436    322184      0.24 111609276   5524800         8

Memory stats from fedora24 from a non-memory related failure.

04:00:00 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
10:10:00 PM   6599268 125316720     95.00  47152416  66153012   1016112      0.75 101375500  12445948     23168
10:20:02 PM   2030148 129885840     98.46  47634616  70201760   3316496      2.44 103386808  14915444         0
10:30:00 PM   6905844 125010144     94.76  47516172  65251180   1437348      1.06 102344020  11170748    227184
10:40:00 PM   7398376 124517612     94.39  49142248  63679960    628060      0.46 102302384  10759868        60
10:50:01 PM   7271360 124644628     94.49  49918100  62994552    736728      0.54 102537956  10655300    138108
11:00:01 PM  14699348 117216640     88.86  48363280  57106708   1138176      0.84  98215676   7589520       420
11:10:00 PM   8108184 123807804     93.85  48387224  62843972   1980452      1.46 100109140  12259052      4056
11:20:01 PM   5045264 126870724     96.18  48497668  66118084   1473176      1.08 101917804  13461764     93568
11:30:00 PM   6089792 125826196     95.38  47711868  65922340   1370312      1.01 102460512  11879964     17480
11:40:01 PM   4283172 127632816     96.75  47737976  67632676   1338792      0.98 103793412  12277640     43176
11:50:01 PM   8365008 123550980     93.66  46810436  64692072   1280548      0.94 101834704  10178976       216

Posting some info while I look into the problem.
Comment 5 Leonardo Sandoval Gonzalez 2017-11-16 16:37:18 UTC 
This is duplicate of bug 12022. This bug has useful memory numbers that are could be useful to solve this issue (also see bug 12022 comment 4).
Comment 6 Leonardo Sandoval Gonzalez 2017-11-16 16:38:05 UTC 
See comment 5 for the reason to mark this as duplicate.

*** This bug has been marked as a duplicate of bug 12022 ***