New issue on the autobuilder, qemux86-tc and qemux86-64-tc builds never finish. It looks like they are stuck on glibc.GlibcSelfTestSystemEmulated.test_glibc. Issue is present on master. First build showing the issue: https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2826 Last successful build: https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2813
We first believed it was just some OOM killer issue inside of the VM, so Richard increased memory size. The issue is still appearing, with various OOM killer / segfault traces but also complaints about NFS. https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/2902 [ 3419.639890] Mem-Info: [ 3419.675310] active_anon:101 inactive_anon:482503 isolated_anon:0 [ 3419.675310] active_file:36 inactive_file:0 isolated_file:0 [ 3419.675310] unevictable:1000 dirty:0 writeback:0 [ 3419.675310] slab_reclaimable:4602 slab_unreclaimable:9191 [ 3419.675310] mapped:14 shmem:2863 pagetables:1322 [ 3419.675310] sec_pagetables:0 bounce:0 [ 3419.675310] kernel_misc_reclaimable:0 [ 3419.675310] free:3300 free_pcp:125 free_cma:0 [ 3419.684054] Node 0 active_anon:404kB inactive_anon:1930012kB active_file:144kB inactive_file:0kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:56kB dirty:0kB writeback:0kB shmem:11452kB writeback_tmp:0kB kernel_stack:2064kB pagetables:5288kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB [ 3419.690399] DMA free:7892kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:7456kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 3419.696619] lowmem_reserve[]: 0 1963 1963 1963 [ 3419.697674] DMA32 free:5308kB boost:0kB min:5640kB low:7644kB high:9648kB reserved_highatomic:0KB free_highatomic:0KB active_anon:404kB inactive_anon:1922556kB active_file:144kB inactive_file:0kB unevictable:4000kB writepending:0kB present:2080608kB managed:2011000kB mlocked:0kB bounce:0kB free_pcp:500kB local_pcp:500kB free_cma:0kB [ 3419.704409] lowmem_reserve[]: 0 0 0 0 [ 3419.705274] DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (UM) 2*64kB (UM) 2*128kB (UM) 1*256kB (U) 2*512kB (UM) 0*1024kB 1*2048kB (U) 1*4096kB (M) = 7892kB [ 3419.708393] DMA32: 174*4kB (UME) 69*8kB (UE) 47*16kB (UE) 32*32kB (UE) 11*64kB (ME) 4*128kB (ME) 0*256kB 2*512kB (ME) 0*1024kB 0*2048kB 0*4096kB = 5264kB [ 3419.711561] 2887 total pagecache pages [ 3419.712451] 0 pages in swap cache [ 3419.713241] Free swap = 0kB [ 3419.713936] Total swap = 0kB [ 3419.714644] 524150 pages RAM [ 3419.715341] 0 pages HighMem/MovableOnly [ 3419.716259] 17560 pages reserved [ 3419.717041] Tasks state (memory values in pages): [ 3419.718156] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 3419.720717] [ 139] 0 139 2692 440 428 12 0 45056 0 -1000 udevd [ 3419.723179] [ 335] 0 335 2379 256 251 5 0 53248 0 -1000 sshd [ 3419.725605] [ 339] 998 339 695 78 34 44 0 40960 0 0 rpcbind [ 3419.728108] [ 345] 997 345 720 33 33 0 0 45056 0 0 rpc.statd [ 3419.730645] [ 374] 0 374 1410 93 75 18 0 45056 0 0 rpc.mountd [ 3419.733219] [ 378] 0 378 1016 14 0 14 0 49152 0 0 syslogd [ 3419.735721] [ 381] 0 381 1016 31 0 31 0 45056 0 0 klogd [ 3419.738185] [ 386] 0 386 1016 28 0 28 0 49152 0 0 start_getty [ 3419.740752] [ 387] 0 387 1016 28 0 28 0 49152 0 0 start_getty [ 3419.743380] [ 388] 0 388 1016 57 0 57 0 49152 0 0 getty [ 3419.745848] [ 391] 0 391 1016 25 0 25 0 40960 0 0 getty [ 3419.748318] [ 392] 0 392 1027 56 32 24 0 36864 0 0 sh [ 3419.750723] [ 31194] 0 31194 2618 320 288 32 0 61440 0 0 sshd-session [ 3419.753359] [ 31196] 0 31196 2683 354 354 0 0 61440 0 0 sshd-session [ 3419.755977] [ 31197] 0 31197 611 21 0 21 0 28672 0 0 ld-linux-x86-64 [ 3419.758650] [ 31198] 0 31198 616679 479140 479104 36 0 3874816 0 0 ld-linux-x86-64 [ 3419.761330] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=ld-linux-x86-64,pid=31198,uid=0 [ 3419.764432] Out of memory: Killed process 31198 (ld-linux-x86-64) total-vm:2466716kB, anon-rss:1916416kB, file-rss:144kB, shmem-rss:0kB, UID:0 pgtables:3784kB oom_score_adj:0 [ 4064.030137] Maximum lock depth 1024 reached task: ld-linux-x86-64 (23759) [ 4690.664501] process 'ld-linux-x86-64' launched '/tmp/testscriptGPVNoi' with NULL argv: empty string added [ 5125.035522] nfs: server 192.168.7.15 not responding, still trying [ 6900.907549] nfs: server 192.168.7.15 not responding, still trying [ 8842.859504] nfs: server 192.168.7.15 not responding, still trying [10642.939461] nfs: server 192.168.7.15 not responding, still trying [12444.651701] nfs: server 192.168.7.15 not responding, still trying [14245.099942] nfs: server 192.168.7.15 not responding, still trying [16069.867826] nfs: server 192.168.7.15 not responding, still trying [17844.395496] nfs: server 192.168.7.15 not responding, still trying [19727.595746] nfs: server 192.168.7.15 not responding, still trying [21445.483456] nfs: server 192.168.7.15 not responding, still trying [23250.156064] nfs: server 192.168.7.15 not responding, still trying [25158.892097] nfs: server 192.168.7.15 not responding, still trying [26903.595825] nfs: server 192.168.7.15 not responding, still trying [28706.028293] nfs: server 192.168.7.15 not responding, still trying https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2862 [ 5518.661575] Mem-Info: [ 5518.741381] active_anon:104 inactive_anon:203410 isolated_anon:0 [ 5518.741381] active_file:11 inactive_file:0 isolated_file:0 [ 5518.741381] unevictable:1000 dirty:0 writeback:0 [ 5518.741381] slab_reclaimable:2688 slab_unreclaimable:6029 [ 5518.741381] mapped:9 shmem:2646 pagetables:281 [ 5518.741381] sec_pagetables:0 bounce:0 [ 5518.741381] kernel_misc_reclaimable:0 [ 5518.741381] free:1727 free_pcp:130 free_cma:0 [ 5518.766782] Node 0 active_anon:416kB inactive_anon:813680kB active_file:24kB inactive_file:28kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:8kB dirty:0kB writeback:0kB shmem:10584kB writeback_tmp:0kB kernel_stack:1024kB pagetables:1124kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB [ 5518.785205] DMA free:3400kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:11960kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 5518.803431] lowmem_reserve[]: 0 834 834 [ 5518.806161] Normal free:3444kB boost:0kB min:3660kB low:4572kB high:5484kB reserved_highatomic:0KB free_highatomic:0KB active_anon:416kB inactive_anon:801740kB active_file:24kB inactive_file:28kB unevictable:4000kB writepending:0kB present:888824kB managed:854440kB mlocked:0kB bounce:0kB free_pcp:520kB local_pcp:20kB free_cma:0kB [ 5518.825619] lowmem_reserve[]: 0 0 0 [ 5518.828064] DMA: 0*4kB 1*8kB (M) 0*16kB 0*32kB 1*64kB (M) 0*128kB 1*256kB (M) 0*512kB 1*1024kB (U) 1*2048kB (U) 0*4096kB = 3400kB [ 5518.835916] Normal: 57*4kB (UME) 24*8kB (UME) 17*16kB (E) 14*32kB (UE) 4*64kB (E) 0*128kB 2*256kB (M) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 3444kB [ 5518.845195] 2659 total pagecache pages [ 5518.847820] 0 pages in swap cache [ 5518.850220] Free swap = 0kB [ 5518.852247] Total swap = 0kB [ 5518.854321] 226204 pages RAM [ 5518.856395] 0 pages HighMem/MovableOnly [ 5518.859084] 8754 pages reserved [ 5518.861304] Tasks state (memory values in pages): [ 5518.864606] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 5518.872064] [ 141] 0 141 2450 271 261 10 0 20480 0 -1000 udevd [ 5518.877785] [ 336] 0 336 2213 225 152 73 0 16384 0 -1000 sshd [ 5518.883389] [ 340] 998 340 766 81 32 49 0 12288 0 0 rpcbind [ 5518.889082] [ 346] 997 346 794 48 33 15 0 12288 0 0 rpc.statd [ 5518.895229] [ 375] 0 375 1502 103 65 38 0 16384 0 0 rpc.mountd [ 5518.900812] [ 379] 0 379 1078 34 0 34 0 12288 0 0 syslogd [ 5518.906300] [ 382] 0 382 1079 77 0 77 0 12288 0 0 klogd [ 5518.911690] [ 387] 0 387 1078 28 0 28 0 12288 0 0 start_getty [ 5518.917457] [ 388] 0 388 1078 71 0 71 0 12288 0 0 start_getty [ 5518.923104] [ 389] 0 389 1079 60 0 60 0 12288 0 0 getty [ 5518.928492] [ 391] 0 391 1079 33 0 33 0 12288 0 0 getty [ 5518.933894] [ 393] 0 393 1089 0 0 0 0 12288 0 0 sh [ 5518.939186] [ 30055] 0 30055 2443 236 192 44 0 20480 0 0 sshd-session [ 5518.944897] [ 30057] 0 30057 2508 263 243 20 0 20480 0 0 sshd-session [ 5518.950734] [ 30058] 0 30058 673 39 0 39 0 8192 0 0 ld-linux.so.2 [ 5518.956492] [ 30059] 0 30059 274500 200770 200736 34 0 815104 0 0 ld-linux.so.2 [ 5518.962270] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=ld-linux.so.2,pid=30059,uid=0 [ 5518.968916] Out of memory: Killed process 30059 (ld-linux.so.2) total-vm:1098000kB, anon-rss:802944kB, file-rss:136kB, shmem-rss:0kB, UID:0 pgtables:796kB oom_score_adj:0 [ 6304.760611] Maximum lock depth 1024 reached task: ld-linux.so.2 (20698) [ 6732.533308] ld-linux.so.2[15529]: segfault at 8d86f2bc ip b7d4bbc7 sp bfeb7b40 error 4 in libc.so[9abc7,b7cd0000+182000] likely on CPU 0 (core 0, socket 0) [ 6732.543180] Code: 14 00 00 00 00 89 d8 66 0f d6 47 08 83 c8 01 89 47 04 89 1c 1f 3b 6c 24 0c 0f 84 2f ff ff ff 8d 6e 08 f7 c5 0f 00 00 00 75 69 <8b> 46 04 8b 4c 24 08 89 c2 c1 ea 03 8d 54 91 04 39 54 24 04 75 72 [ 7269.314336] process 'ld-linux.so.2' launched '/tmp/testscriptTwrT96' with NULL argv: empty string added [ 7924.284321] nfs: server 192.168.7.13 not responding, still trying [ 9693.052459] nfs: server 192.168.7.13 not responding, still trying [11747.132312] nfs: server 192.168.7.13 not responding, still trying [13550.844442] nfs: server 192.168.7.13 not responding, still trying [15414.525119] nfs: server 192.168.7.13 not responding, still trying [17257.724924] nfs: server 192.168.7.13 not responding, still trying [18982.140799] nfs: server 192.168.7.13 not responding, still trying [20800.764720] nfs: server 192.168.7.13 not responding, still trying [22598.908290] nfs: server 192.168.7.13 not responding, still trying [24454.396766] nfs: server 192.168.7.13 not responding, still trying [26182.908793] nfs: server 192.168.7.13 not responding, still trying [27952.380437] nfs: server 192.168.7.13 not responding, still trying [29787.389301] nfs: server 192.168.7.13 not responding, still trying [31651.069794] nfs: server 192.168.7.13 not responding, still trying [33432.829107] nfs: server 192.168.7.13 not responding, still trying [35255.550111] nfs: server 192.168.7.13 not responding, still trying [37131.517850] nfs: server 192.168.7.13 not responding, still trying [38872.317280] nfs: server 192.168.7.13 not responding, still trying [40610.941152] nfs: server 192.168.7.13 not responding, still trying [42476.798778] nfs: server 192.168.7.13 not responding, still trying [44397.820804] nfs: server 192.168.7.13 not responding, still trying [46073.086684] nfs: server 192.168.7.13 not responding, still trying [47854.847185] nfs: server 192.168.7.13 not responding, still trying [49681.662610] nfs: server 192.168.7.13 not responding, still trying [51639.549796] nfs: server 192.168.7.13 not responding, still trying [53290.237524] nfs: server 192.168.7.13 not responding, still trying [55108.863194] nfs: server 192.168.7.13 not responding, still trying https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/2904 [ 75.413237] block nvme0n1: No UUID available providing old NGUID [50829.486190] nvme 0000:c2:00.0: Using 64-bit DMA addresses [50869.447840] nvme 0000:c4:00.0: Using 64-bit DMA addresses [218227.405389] perf: interrupt took too long (2732 > 2500), lowering kernel.perf_event_max_sample_rate to 73000 [226051.505453] hrtimer: interrupt took 5140 ns [288121.214431] rustc[2304282]: segfault at 7fddfc5fef58 ip 00007fde039a4cd4 sp 00007fddfc5fee30 error 6 in librustc_driver-d8d270262fe3a7b2.so[7fddfe1e1000+6001000] likely on CPU 1 (core 1, socket 0) [288121.214483] Code: 7c 24 70 e8 3e 3c fe ff 48 89 df e8 b6 cf 83 fa 66 0f 1f 44 00 00 55 41 57 41 56 41 55 41 54 53 48 81 ec c8 02 00 00 48 89 cd <48> 89 94 24 28 01 00 00 49 89 f5 49 89 fe 4c 8d 67 08 48 c7 84 24 [288122.659830] rustc[2305007]: segfault at 7fd85a1fef30 ip 00007fd86008ba24 sp 00007fd85a1fef10 error 6 in librustc_driver-d8d270262fe3a7b2.so[7fd85bde1000+6001000] likely on CPU 34 (core 60, socket 0) [288122.659890] Code: 00 48 89 df ff 15 c4 33 06 06 4c 89 f7 e8 64 62 d5 fb 0f 1f 40 00 55 41 57 41 56 41 55 41 54 53 48 81 ec 68 01 00 00 49 89 f7 <48> 89 7c 24 20 48 8b 1e 4c 8b 76 08 49 8b 86 d8 00 00 00 48 8b 88 [288126.385936] rustc[2303417]: segfault at 7f209a5fefa0 ip 00007f20a129bba9 sp 00007f209a5fef90 error 6 in librustc_driver-d8d270262fe3a7b2.so[7f209c1e1000+6001000] likely on CPU 87 (core 11, socket 0) [288126.385997] Code: 00 00 48 89 b4 24 c8 00 00 00 48 39 ca 0f 85 6e 06 00 00 49 89 d7 49 89 f6 48 89 fb 0f b6 86 f5 00 00 00 0f b6 8e f7 00 00 00 <48> 89 54 24 10 88 4c 24 18 88 44 24 19 48 8d 74 24 10 4c 89 f7 e8 [288159.028923] rustc[2453136]: segfault at 7f7678dfef58 ip 00007f76801a4cd4 sp 00007f7678dfee30 error 6 in librustc_driver-d8d270262fe3a7b2.so[7f767a9e1000+6001000] likely on CPU 86 (core 10, socket 0) [288159.028981] Code: 7c 24 70 e8 3e 3c fe ff 48 89 df e8 b6 cf 83 fa 66 0f 1f 44 00 00 55 41 57 41 56 41 55 41 54 53 48 81 ec c8 02 00 00 48 89 cd <48> 89 94 24 28 01 00 00 49 89 f5 49 89 fe 4c 8d 67 08 48 c7 84 24 [288160.174738] rustc[2453961]: segfault at 7fd884bfef30 ip 00007fd88aa8ba24 sp 00007fd884bfef10 error 6 in librustc_driver-d8d270262fe3a7b2.so[7fd8867e1000+6001000] likely on CPU 21 (core 51, socket 0) [288160.174785] Code: 00 48 89 df ff 15 c4 33 06 06 4c 89 f7 e8 64 62 d5 fb 0f 1f 40 00 55 41 57 41 56 41 55 41 54 53 48 81 ec 68 01 00 00 49 89 f7 <48> 89 7c 24 20 48 8b 1e 4c 8b 76 08 49 8b 86 d8 00 00 00 48 8b 88 [288163.318619] rustc[2452447]: segfault at 7f6725dfefa0 ip 00007f672ca9bba9 sp 00007f6725dfef90 error 6 in librustc_driver-d8d270262fe3a7b2.so[7f67279e1000+6001000] likely on CPU 11 (core 37, socket 0) [288163.318681] Code: 00 00 48 89 b4 24 c8 00 00 00 48 39 ca 0f 85 6e 06 00 00 49 89 d7 49 89 f6 48 89 fb 0f b6 86 f5 00 00 00 0f b6 8e f7 00 00 00 <48> 89 54 24 10 88 4c 24 18 88 44 24 19 48 8d 74 24 10 4c 89 f7 e8 [290206.755690] process 'qemu-riscv64' launched '/tmp/testscriptfWbQ0x' with NULL argv: empty string added https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2864 [ 3282.273312] workqueue: vmstat_update hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND [ 3282.274516] workqueue: vmstat_update hogged CPU for >10000us 11 times, consider switching to WQ_UNBOUND [50746.406751] workqueue: mmput_async_fn hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND [50751.093766] workqueue: mmput_async_fn hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND [51016.480293] workqueue: mmput_async_fn hogged CPU for >10000us 11 times, consider switching to WQ_UNBOUND [51943.067799] workqueue: mmput_async_fn hogged CPU for >10000us 19 times, consider switching to WQ_UNBOUND [51943.088670] workqueue: mmput_async_fn hogged CPU for >10000us 35 times, consider switching to WQ_UNBOUND [51944.835683] workqueue: mmput_async_fn hogged CPU for >10000us 67 times, consider switching to WQ_UNBOUND [51948.716684] workqueue: mmput_async_fn hogged CPU for >10000us 131 times, consider switching to WQ_UNBOUND [51956.513826] workqueue: mmput_async_fn hogged CPU for >10000us 259 times, consider switching to WQ_UNBOUND [51964.525868] workqueue: mmput_async_fn hogged CPU for >10000us 515 times, consider switching to WQ_UNBOUND [51976.776863] workqueue: mmput_async_fn hogged CPU for >10000us 1027 times, consider switching to WQ_UNBOUND [51987.906464] show_signal_msg: 179 callbacks suppressed [51987.906469] rustc[3970818]: segfault at 7693aabfeec8 ip 00007693b1f83ec4 sp 00007693aabfedb0 error 6 in librustc_driver-2563db0499643e31.so[6183ec4,7693ac7c0000+5ff0000] likely on CPU 11 (core 37, socket 0) [51987.906482] Code: 7c 24 70 e8 5e 3b fe ff 48 89 df e8 26 cb 83 fa 66 0f 1f 44 00 00 55 41 57 41 56 41 55 41 54 53 48 81 ec b8 02 00 00 48 89 cd <48> 89 94 24 18 01 00 00 49 89 f5 49 89 fe 48 8d 5f 08 48 c7 84 24 [51988.691646] rustc[3971310]: segfault at 779e9f7fff20 ip 0000779ea9a9d114 sp 0000779e9f7fff00 error 6 in librustc_driver-2563db0499643e31.so[4c9d114,779ea57c0000+5ff0000] likely on CPU 27 (core 27, socket 0) [51988.691659] Code: e8 a1 c8 ff ff 48 89 df e8 d9 38 d2 fb ff 15 1b 7b 00 06 0f 1f 00 55 41 57 41 56 41 55 41 54 53 48 81 ec 68 01 00 00 49 89 f7 <48> 89 7c 24 20 48 8b 1e 4c 8b 76 08 49 8b 86 d8 00 00 00 48 8b 88 [51990.779034] rustc[3970507]: segfault at 71e1a29feff8 ip 000071e1a9645271 sp 000071e1a29fef30 error 6 in librustc_driver-2563db0499643e31.so[5a45271,71e1a45c0000+5ff0000] likely on CPU 54 (core 32, socket 0) [51990.779048] Code: 7e a6 90 03 48 8d 15 5e ef 13 05 be 31 00 00 00 ff 15 a3 b9 25 05 0f 1f 00 55 41 57 41 56 41 55 41 54 53 48 81 ec 48 01 00 00 <48> 89 94 24 c8 00 00 00 48 89 8c 24 d0 00 00 00 48 89 b4 24 d8 00 [51996.777003] workqueue: mmput_async_fn hogged CPU for >10000us 2051 times, consider switching to WQ_UNBOUND [53431.909252] process 'qemu-riscv64' launched '/tmp/testscriptsX4hNW' with NULL argv: empty string added [62708.775181] NFS: v4 server nas-vk.int.yocto.io does not accept raw uid/gids. Reenabling the idmapper. [216562.876228] perf: interrupt took too long (2794 > 2500), lowering kernel.perf_event_max_sample_rate to 71000 [222744.171206] nvme 0000:c4:00.0: Using 64-bit DMA addresses [224085.356252] workqueue: delayed_vfree_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [224085.358330] workqueue: delayed_vfree_work hogged CPU for >10000us 5 times, consider switching to WQ_UNBOUND [224162.042586] workqueue: delayed_vfree_work hogged CPU for >10000us 7 times, consider switching to WQ_UNBOUND [224352.754153] workqueue: delayed_vfree_work hogged CPU for >10000us 11 times, consider switching to WQ_UNBOUND [287132.945448] nvme 0000:c2:00.0: Using 64-bit DMA addresses [287184.531920] workqueue: delayed_vfree_work hogged CPU for >10000us 19 times, consider switching to WQ_UNBOUND [287370.262257] hrtimer: interrupt took 1460 ns [287695.994301] workqueue: mmput_async_fn hogged CPU for >10000us 4099 times, consider switching to WQ_UNBOUND [287723.281320] rustc[4110944]: segfault at 7efd14ffef58 ip 00007efd1c3a62b4 sp 00007efd14ffee30 error 6 in librustc_driver-d8d270262fe3a7b2.so[61a62b4,7efd16be1000+6002000] likely on CPU 31 (core 57, socket 0) [287723.281335] Code: 7c 24 70 e8 3e 3c fe ff 48 89 df e8 36 b7 83 fa 66 0f 1f 44 00 00 55 41 57 41 56 41 55 41 54 53 48 81 ec c8 02 00 00 48 89 cd <48> 89 94 24 28 01 00 00 49 89 f5 49 89 fe 4c 8d 67 08 48 c7 84 24 [287724.115356] rustc[4111627]: segfault at 774d723fef30 ip 0000774d7828d004 sp 0000774d723fef10 error 6 in librustc_driver-d8d270262fe3a7b2.so[4c8d004,774d73fe1000+6002000] likely on CPU 91 (core 41, socket 0) [287724.115370] Code: 00 48 89 df ff 15 dc 1d 06 06 4c 89 f7 e8 e4 49 d5 fb 0f 1f 40 00 55 41 57 41 56 41 55 41 54 53 48 81 ec 68 01 00 00 49 89 f7 <48> 89 7c 24 20 48 8b 1e 4c 8b 76 08 49 8b 86 d8 00 00 00 48 8b 88 [287726.401289] rustc[4110598]: segfault at 770dca3fefa0 ip 0000770dd109d189 sp 0000770dca3fef90 error 6 in librustc_driver-d8d270262fe3a7b2.so[5a9d189,770dcbfe1000+6002000] likely on CPU 36 (core 8, socket 0) [287726.401302] Code: 00 00 48 89 b4 24 c8 00 00 00 48 39 ca 0f 85 6e 06 00 00 49 89 d7 49 89 f6 48 89 fb 0f b6 86 f5 00 00 00 0f b6 8e f7 00 00 00 <48> 89 54 24 10 88 4c 24 18 88 44 24 19 48 8d 74 24 10 4c 89 f7 e8 [287736.580698] workqueue: mmput_async_fn hogged CPU for >10000us 8195 times, consider switching to WQ_UNBOUND
First build in my previous message is failing on rust.RustSelfTestSystemEmulated.test_rust, not gcc.GccCrossSelfTestSystemEmulated.test_cross_gcc. So a different test, but still a QEMU related issue.
Two new failures this morning, I will attach the complete boot logs in separate message, but two notes. First, it turns out I was reading the log wrong the whole time, the failing test is probably glibc.GlibcSelfTestSystemEmulated.test_glibc. Command running inside of QEMU is in glibc testsuite: - on ubuntu2204-vk-4: /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-1688574/tmp/work/core2-32-poky-linux/glibc-testsuite/2.42+git/build-i686-poky-linux/socket/tst-socket-timestamp-compat-time64 - on debian11-vk-1: /srv/pokybuild/yocto-worker/qemux86-64-tc/build/build-st-303450/tmp/work/x86-64-v3-poky-linux/glibc-testsuite/2.42+git/build-x86_64-poky-linux/sysvipc/test-sysvmsg Richard read this correctly and did increase the QEMU memory for the correct test. Also, while we do have OOM killer triggered, we can see some NFS errors way before, so maybe OOM killer is some consequence and not the origin of the issue: [ 2.661598] NFSD: Unable to initialize client recovery tracking! (-110) [ 2.662617] NFSD: Is nfsdcld running? If not, enable CONFIG_NFSD_LEGACY_CLIENT_TRACKING. [ 2.663699] NFSD: starting 90-second grace period (net f0000000) ... [ 4041.468837] Out of memory: Killed process 31277 (ld-linux-x86-64) total-vm:2466716kB, anon-rss:1915264kB, file-rss:16kB, shmem-rss:0kB, UID:0 pgtables:3788kB oom_score_adj:0 ... [ 5827.637739] nfs: server 192.168.7.15 not responding, still trying
Created attachment 5164 [details] qemu boot log qemux86-tc ubuntu2204-vk-4 https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2869
Created attachment 5165 [details] qemu boot log debian11-vk-1 qemux86-64-tc https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/2909
qemux86-64-tc stream9-vk-1 master completed at 2026-01-02 16:09:47+00:00 https://valkyrie.yocto.io/pub/non-release/20260102-8/testresults/qemux86-64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/2920/steps/13/logs/stdio
I suspect this one is a variant but still related with this issue. qemuarm-tc stream9-vk-1 master completed at 2026-01-04 02:14:00+00:00 https://valkyrie.yocto.io/pub/non-release/20260104-8/testresults/qemuarm-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/42/builds/2898/steps/13/logs/stdio
Hopefully fixed with 'Revert "glibc: Enable NFS local file locking for glibc tests"'.
Richard reproduced this on his older Ubuntu distro on his laptop. WindRiver folks have not yet been able to reproduce except on the YP AB. It's frustrating that it takes 4 hours to reproduce, It would be nice if we could reduce that time. Also the logging disappears which isn't ideal. Sundeep, please update our status every day even if you haven't been reproduced. IRC chat would also be good.
I was able to get some output from the unfs command when it fails: Command '['unfsd', '-d', '-p', '-e', '/tmp/tmpbs3p6e7v', '-n', '37583', '-m', '40459']' returned -6 as exit code. Last 20 lines: UNFS3 unfsd 0.11.0 (C) 2003-2025, various authors /media/build/poky/build/build-st-1631462/tmp/work: ip ::/0 options 21 *** buffer overflow detected ***: terminated so the issue is unfsd failing and it terminates due to a buffer overflow. What causes this and how we can make the glibc tests exit earlier remains to be seen but it gives us a thread to pull on.
This ugly hacking branch gives me a stackdump of unfsd: https://git.openembedded.org/openembedded-core-contrib/log/?h=adrianf/unfsd-stackdumps It adds an opt-in segfault feature to unfsd which allows me to test it (Running: ['unfsd', '-S'...] where -S means segfault on purpose). This looks like: rm -rf build-st*; oe-selftest -v -r glibc.GlibcSelfTestSystemEmulated.test_glibc 2026-01-10 16:37:18,855 - oe-selftest - INFO - Changing cwd to /home/adrian/projets/oss/oe/bitbake-builds/poky-master/build 2026-01-10 16:37:18,855 - oe-selftest - INFO - Adding layer libraries: 2026-01-10 16:37:18,855 - oe-selftest - INFO - /home/adrian/projets/oss/oe/bitbake-builds/poky-master/layers/meta-yocto/meta-poky/lib 2026-01-10 16:37:18,855 - oe-selftest - INFO - /home/adrian/projets/oss/oe/bitbake-builds/poky-master/layers/openembedded-core/meta/lib 2026-01-10 16:37:18,855 - oe-selftest - INFO - /home/adrian/projets/oss/oe/bitbake-builds/poky-master/layers/meta-yocto/meta-yocto-bsp/lib 2026-01-10 16:37:18,855 - oe-selftest - INFO - /home/adrian/projets/oss/oe/bitbake-builds/poky-master/layers/openembedded-core/meta-selftest/lib 2026-01-10 16:37:18,856 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2026-01-10 16:37:19,523 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/adrian/projets/oss/oe/bitbake-builds/poky-master/build-st/conf/local.conf 2026-01-10 16:37:19,523 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2026-01-10 16:37:19,523 - oe-selftest - INFO - test_glibc (glibc.GlibcSelfTestSystemEmulated.test_glibc) 2026-01-10 16:37:42,419 - oe-selftest - INFO - Running: ['unfsd', '-S', '-d', '-p', '-e', '/tmp/tmp8p6cen9f', '-n', '46553', '-m', '48229'] 2026-01-10 16:37:42,419 - oe-selftest - DEBUG - Writing to: /home/adrian/projets/oss/oe/bitbake-builds/poky-master/build-st/conf/selftest.inc IMAGE_FEATURES += "ssh-server-openssh" CORE_IMAGE_EXTRA_INSTALL += "glibc-charmaps libgcc libstdc++ libatomic libgomp nfs-utils" 2026-01-10 16:37:42,419 - oe-selftest - INFO - UNFS3 unfsd 0.11.0 (C) 2003-2025, various authors 2026-01-10 16:37:42,422 - oe-selftest - INFO - /home/adrian/projets/oss/oe/bitbake-builds/poky-master/build-st/tmp/work: ip ::/0 options 21 2026-01-10 16:37:42,422 - oe-selftest - INFO - Testing segmentation fault... 2026-01-10 16:37:42,422 - oe-selftest - INFO - Segmentation fault 2026-01-10 16:37:42,422 - oe-selftest - INFO - Backtrace: 2026-01-10 16:37:42,422 - oe-selftest - INFO - 0: daemon_exit (ip=0x402c07) 2026-01-10 16:37:42,422 - oe-selftest - INFO - 1: (ip=0x7f58cea80ba0) 2026-01-10 16:37:42,422 - oe-selftest - INFO - 2: main (ip=0x403b25) 2026-01-10 16:37:42,422 - oe-selftest - INFO - 3: (ip=0x7f58cea6af68) 2026-01-10 16:37:42,422 - oe-selftest - INFO - 4: __libc_start_main (ip=0x7f58cea6b01b) 2026-01-10 16:37:42,422 - oe-selftest - INFO - 5: _start (ip=0x400be5) Now I'm going to run the same branch with the -S commit reverted. If I can get a stack trace like that, it's probably the one we are searching for. Not sure what I can get over the weekend. A full, successful oe-selftest -r glibc.GlibcSelfTestSystemEmulated.test_glibc test run takes ~8 hours on my machine. Running only tst-syslog should run much faster, I need to check... Also an issue is, that the test does not terminate, when unfsd crashes.
We tried several builds on our local with ab sstate cache & j32, but all the time the builds are success. Once I tried by increasing the qemu mem to 4096 & qemu param with 'kvm', with that build completed in 11hrs. And, w/o these build took 25hrs to complete. During test run I see the below unfsd process is running but not terminated with OOM or segfault. unfsd -d -p -e /tmp/tmpri21v8bs -n 43229 -m 54277 And the mem usage of this process also shows normal. I tried triggering the test on ab by increasing the qemu mem with kvm but the qemux86-64-tc builder seems not working and unable to trigger any test during weekend.
https://lists.openembedded.org/g/openembedded-core/message/229209
My link has details on how to just run the specific faulting test
I've confirmed the message is from debug/chk_fail.c in glibc which calls debug/fortify_fail.c
qemux86-tc ubuntu2204-vk-3 master completed at 2026-01-12 10:17:20+00:00 https://valkyrie.yocto.io/pub/non-release/20260112-28/testresults/qemux86-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2939/steps/13/logs/stdio qemuarm64-tc opensuse156-vk-1 mathieu/master-next completed at 2026-01-09 10:27:05+00:00 https://valkyrie.yocto.io/pub/non-release/20260109-59/testresults/qemuarm64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/5/builds/2938/steps/14/logs/stdio qemuriscv64-tc rocky8-vk-1 mathieu/master-next completed at 2026-01-09 10:27:06+00:00 https://valkyrie.yocto.io/pub/non-release/20260109-59/testresults/qemuriscv64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/58/builds/899/steps/14/logs/stdio qemux86-64-tc debian12-vk-8 mathieu/master-next completed at 2026-01-09 10:27:06+00:00 https://valkyrie.yocto.io/pub/non-release/20260109-59/testresults/qemux86-64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/2984/steps/13/logs/stdio qemux86-tc debian13-vk-2 mathieu/master-next completed at 2026-01-09 10:27:06+00:00 https://valkyrie.yocto.io/pub/non-release/20260109-59/testresults/qemux86-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2932/steps/13/logs/stdio qemux86-64-tc fedora40-vk-3 master completed at 2026-01-12 10:17:20+00:00 https://valkyrie.yocto.io/pub/non-release/20260112-28/testresults/qemux86-64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/2999/steps/13/logs/stdio qemux86-tc ubuntu2204-vk-1 master completed at 2026-01-12 10:17:36+00:00 https://valkyrie.yocto.io/pub/non-release/20260112-27/testresults/qemux86-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2940/steps/13/logs/stdio qemux86-64-tc fedora40-vk-4 master completed at 2026-01-12 10:17:36+00:00 https://valkyrie.yocto.io/pub/non-release/20260112-27/testresults/qemux86-64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/3000/steps/13/logs/stdio qemux86-64-tc fedora41-vk-1 master completed at 2026-01-12 10:18:05+00:00 https://valkyrie.yocto.io/pub/non-release/20260112-30/testresults/qemux86-64-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/3001/steps/13/logs/stdio qemux86-tc alma8-vk-1 master completed at 2026-01-12 10:18:07+00:00 https://valkyrie.yocto.io/pub/non-release/20260112-30/testresults/qemux86-tc/ https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2941/steps/14/logs/stdio
https://git.openembedded.org/openembedded-core/commit/?id=e51d5e19cb1ba1d5ad7442064b64821d178bc9ca