14673 2022-01-06 13:53:14 +0000 stap failure on core-image-sato-sdk for 5.10 on qemuarm 2022-02-10 15:54:39 +0000 1 1 7 Build System, Metadata & Runtime OE-Core kernel 4.0 x86 Multiple RESOLVED FIXED AB-INT High normal 4.0 M3 1 richard.purdie ross.burton alexandre.belloni bruce.ashfield jon.mason randy.macleod ross.burton tom.zanussi --- --- No (bug/feature does not impact docs) oldest_to_newest 92217 0 richard.purdie 2022-01-06 13:53:14 +0000 We're seeing a lot of qemu crashes when running the systemtap tests for qemuarm. It is with poky-altcfg and hence the 5.10 kernel is likely at fault. https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3443/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3434/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3432/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3430/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3426/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3397/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3395/steps/13/logs/stdio Kernel stack trace is in this saved log: http://autobuilder.yocto.io/pub/failed-builds-data/tumbleweed-ty-3/qemuarm-stap/qemu_boot_log.20220106024610 testimage log: http://autobuilder.yocto.io/pub/failed-builds-data/tumbleweed-ty-3/qemuarm-stap/log.do_testimage.16806 (from build 3443 above) 92245 1 richard.purdie 2022-01-07 14:57:29 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3449/steps/13/logs/stdio 92296 2 alexandre.belloni 2022-01-13 08:42:11 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3458/steps/13/logs/stdio AssertionError: 255 != 0 : SYSTEMD_BUS_TIMEOUT=240s systemctl --version ssh: connect to host 192.168.7.6 port 22: No route to host AssertionError: 255 != 0 : Xorg does not appear to be running ssh: connect to host 192.168.7.6 port 22: No route to host qemuarm-alt fedora35-ty-1 fedora35-ty-1 has an SSD https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3460/steps/14/logs/stdio qemuarm-alt debian9-ty-2 92297 3 alexandre.belloni 2022-01-13 08:45:32 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3848/steps/16/logs/stdio qemuarm64-armhost ubuntu1804-arm-1 92301 4 alexandre.belloni 2022-01-13 10:31:10 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3465/steps/13/logs/stdio qemuarm-alt stream8-ty-1 92307 5 alexandre.belloni 2022-01-13 11:58:25 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3470/steps/13/logs/stdio qemuarm-alt alma8-ty-2 92308 6 alexandre.belloni 2022-01-13 12:01:59 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3479/steps/14/logs/stdio qemuarm-alt centos7-ty-4 https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3934/steps/16/logs/stdio qemuarm64-armhost ubuntu2004-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3483/steps/13/logs/stdio qemuarm-alt debian11-ty-3 92338 7 alexandre.belloni 2022-01-17 15:41:01 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3490/steps/13/logs/stdio 92343 8 alexandre.belloni 2022-01-17 16:26:59 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3944/steps/16/logs/stdio 92362 9 alexandre.belloni 2022-01-20 15:34:23 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3512/steps/13/logs/stdio 92391 10 ross.burton 2022-01-24 17:54:17 +0000 Putting a recent dmesg inline for ease of reference: [ 2653.988532] connmand invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [ 2653.996964] CPU: 1 PID: 196 Comm: connmand Tainted: G O 5.10.91-yocto-standard #1 [ 2653.997541] Hardware name: linux,dummy-virt (DT) [ 2653.998447] Call trace: [ 2653.999486] dump_backtrace+0x0/0x1a0 [ 2654.000097] show_stack+0x24/0x30 [ 2654.000660] dump_stack+0xd4/0x110 [ 2654.000922] dump_header+0x50/0x1fc [ 2654.001385] oom_kill_process+0x1f4/0x1fc [ 2654.001734] out_of_memory+0xf0/0x340 [ 2654.001952] __alloc_pages_nodemask+0xa24/0xe90 [ 2654.002227] pagecache_get_page+0x114/0x330 [ 2654.002624] filemap_fault+0x6d0/0x87c [ 2654.003022] ext4_filemap_fault+0x40/0xf0 [ 2654.003457] __do_fault+0x44/0x10c [ 2654.003726] handle_mm_fault+0x6d8/0xc20 [ 2654.004075] do_page_fault+0x15c/0x400 [ 2654.004309] do_translation_fault+0xbc/0xd8 [ 2654.004594] do_mem_abort+0x4c/0xac [ 2654.005309] el1_abort+0x44/0x6c [ 2654.005864] el1_sync_handler+0xb0/0xd0 [ 2654.006199] el1_sync+0x88/0x140 [ 2654.006516] strncpy_from_user+0x17c/0x250 [ 2654.006769] user_path_at_empty+0x3c/0x70 [ 2654.006997] vfs_statx+0x90/0x164 [ 2654.007218] __do_sys_newfstatat+0x34/0x74 [ 2654.007429] __arm64_sys_newfstatat+0x2c/0x3c [ 2654.008011] el0_svc_common.constprop.0+0x9c/0x1f0 [ 2654.008260] do_el0_svc+0x7c/0xa0 [ 2654.008595] el0_svc+0x20/0x30 [ 2654.008855] el0_sync_handler+0xa4/0x130 [ 2654.009255] el0_sync+0x180/0x1c0 [ 2654.009938] Mem-Info: [ 2654.010541] active_anon:326 inactive_anon:98257 isolated_anon:0 [ 2654.010541] active_file:397 inactive_file:420 isolated_file:0 [ 2654.010541] unevictable:768 dirty:0 writeback:0 [ 2654.010541] slab_reclaimable:4046 slab_unreclaimable:7944 [ 2654.010541] mapped:1920 shmem:3587 pagetables:942 bounce:0 [ 2654.010541] free:1550 free_pcp:115 free_cma:0 [ 2654.012563] Node 0 active_anon:1304kB inactive_anon:393028kB active_file:1588kB inactive_file:1832kB unevictable:3072kB isolated(anon):0kB isolated(file):0kB mapped:7832kB dirty:0kB writeback:0kB shmem:14348kB writeback_tmp:0kB kernel_stack:2272kB all_unreclaimable? no [ 2654.014107] DMA free:5952kB min:2784kB low:3480kB high:4176kB reserved_highatomic:0KB active_anon:1304kB inactive_anon:393028kB active_file:1792kB inactive_file:2028kB unevictable:3072kB writepending:0kB present:524288kB managed:488312kB mlocked:0kB pagetables:3768kB bounce:0kB free_pcp:432kB local_pcp:0kB free_cma:0kB [ 2654.015485] lowmem_reserve[]: 0 0 0 0 [ 2654.016701] DMA: 240*4kB (UME) 217*8kB (UME) 77*16kB (UME) 37*32kB (UE) 10*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5752kB [ 2654.017644] 4569 total pagecache pages [ 2654.018008] 0 pages in swap cache [ 2654.018265] Swap cache stats: add 0, delete 0, find 0/0 [ 2654.018524] Free swap = 0kB [ 2654.018759] Total swap = 0kB [ 2654.019222] 131072 pages RAM [ 2654.019420] 0 pages HighMem/MovableOnly [ 2654.019881] 8994 pages reserved [ 2654.020302] Tasks state (memory values in pages): [ 2654.020544] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 2654.022396] [ 128] 998 128 1168 42 32768 0 0 rpcbind [ 2654.023029] [ 129] 0 129 24870 668 61440 0 -250 systemd-journal [ 2654.023596] [ 149] 0 149 3583 178 40960 0 -1000 systemd-udevd [ 2654.024240] [ 174] 0 174 75698 407 69632 0 0 rngd [ 2654.024664] [ 181] 0 181 852 39 32768 0 0 klogd [ 2654.025073] [ 184] 0 184 715 63 32768 0 0 crond [ 2654.025373] [ 185] 999 185 1441 193 32768 0 -900 dbus-daemon [ 2654.025895] [ 188] 0 188 2508 191 45056 0 0 ofonod [ 2654.026187] [ 190] 0 190 2342 155 40960 0 0 systemd-logind [ 2654.026680] [ 196] 0 196 3365 194 49152 0 0 connmand [ 2654.026972] [ 198] 995 198 2651 163 45056 0 0 systemd-network [ 2654.027355] [ 206] 997 206 1482 95 36864 0 0 avahi-daemon [ 2654.027964] [ 207] 997 207 1437 65 36864 0 0 avahi-daemon [ 2654.028484] [ 211] 0 211 2868 172 49152 0 0 wpa_supplicant [ 2654.029050] [ 219] 0 219 8393 248 49152 0 0 tcf-agent [ 2654.029594] [ 220] 0 220 1300 20 32768 0 0 agetty [ 2654.029914] [ 221] 0 221 581 29 28672 0 0 agetty [ 2654.030240] [ 222] 0 222 1321 102 32768 0 0 login [ 2654.030526] [ 229] 0 229 2220 95 40960 0 0 systemd-userdbd [ 2654.031158] [ 231] 0 231 984 34 32768 0 0 xinit [ 2654.031526] [ 237] 0 237 30637 3311 126976 0 0 Xorg [ 2654.032164] [ 243] 0 243 2743 273 45056 0 100 systemd [ 2654.032417] [ 244] 0 244 3373 571 53248 0 100 (sd-pam) [ 2654.032699] [ 249] 0 249 2277 231 45056 0 0 matchbox-window [ 2654.033009] [ 255] 0 255 1692 90 36864 0 0 dbus-launch [ 2654.033412] [ 256] 0 256 1352 90 36864 0 0 dbus-daemon [ 2654.033644] [ 257] 0 257 1251 354 36864 0 0 sh [ 2654.033915] [ 267] 0 267 57824 183 65536 0 0 at-spi-bus-laun [ 2654.034371] [ 274] 0 274 54997 1764 151552 0 0 connman-applet [ 2654.034859] [ 296] 0 296 1325 58 32768 0 0 dbus-daemon [ 2654.035276] [ 299] 0 299 48618 2074 135168 0 0 matchbox-deskto [ 2654.035893] [ 300] 0 300 50040 1490 135168 0 0 matchbox-panel [ 2654.036231] [ 303] 0 303 38364 326 65536 0 0 pulseaudio [ 2654.036532] [ 305] 0 305 2862 127 49152 0 0 gconfd-2 [ 2654.036852] [ 308] 0 308 10401 785 86016 0 0 settings-daemon [ 2654.037178] [ 321] 0 321 40247 186 65536 0 0 at-spi2-registr [ 2654.037646] [ 11081] 993 11081 20764 115 49152 0 0 systemd-timesyn [ 2654.038074] [ 18152] 0 18152 852 23 32768 0 0 syslogd [ 2654.038419] [ 18501] 0 18501 2247 104 40960 0 0 systemd-userwor [ 2654.038936] [ 18507] 0 18507 2247 103 40960 0 0 systemd-userwor [ 2654.039251] [ 18724] 0 18724 2222 87 40960 0 0 systemd-userwor [ 2654.039566] [ 19067] 0 19067 2716 366 40960 0 0 sshd [ 2654.040104] [ 19070] 0 19070 21404 16901 196608 0 0 stap [ 2654.040408] [ 19071] 0 19071 705 151 32768 0 0 make [ 2654.040671] [ 19300] 0 19300 733 157 32768 0 0 make [ 2654.040938] [ 19320] 0 19320 733 195 28672 0 0 make [ 2654.041224] [ 19514] 0 19514 963 86 32768 0 0 sh [ 2654.041488] [ 19515] 0 19515 892 21 28672 0 0 gcc [ 2654.041752] [ 19516] 0 19516 23584 15126 212992 0 0 cc1 [ 2654.042042] [ 19517] 0 19517 963 87 32768 0 0 sh [ 2654.042318] [ 19518] 0 19518 892 22 28672 0 0 gcc [ 2654.042598] [ 19519] 0 19519 22877 14504 204800 0 0 cc1 [ 2654.042955] [ 19520] 0 19520 963 86 28672 0 0 sh [ 2654.043220] [ 19521] 0 19521 963 85 28672 0 0 sh [ 2654.043474] [ 19522] 0 19522 892 21 28672 0 0 gcc [ 2654.044065] [ 19523] 0 19523 22299 13866 200704 0 0 cc1 [ 2654.044332] [ 19524] 0 19524 892 22 28672 0 0 gcc [ 2654.063196] [ 19525] 0 19525 22306 13780 204800 0 0 cc1 [ 2654.064216] [ 19532] 0 19532 963 86 36864 0 0 sh [ 2654.064715] [ 19533] 0 19533 892 22 28672 0 0 gcc [ 2654.065371] [ 19534] 0 19534 14560 6086 135168 0 0 cc1 [ 2654.066588] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-c190.scope,task=stap,pid=19070,uid=0 [ 2654.070241] Out of memory: Killed process 19070 (stap) total-vm:85616kB, anon-rss:67572kB, file-rss:32kB, shmem-rss:0kB, UID:0 pgtables:192kB oom_score_adj:0 [ 2654.189111] oom_reaper: reaped process 19070 (stap), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB OOM killer steps in and decides that stap is the one to go (as it's the parent of all the GCC instances). It is eating a fair bit of memory, but not crazy amounts. 92401 11 alexandre.belloni 2022-01-26 23:07:09 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/131/builds/39/steps/13/logs/stdio 92405 12 alexandre.belloni 2022-01-26 23:28:12 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/4653/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3528/steps/13/logs/stdio This is now: AssertionError: 1 != 0 : stap -v --disable-cache -DSTP_NO_VERREL_CHECK -e 'probe oneshot { print("Hello, "); println("world!") }' WARNING: Kernel function symbol table missing [man warning::symbols] Pass 1: parsed user script and 463 library scripts using 50452virt/43716res/6568shr/37408data kb, in 3710usr/810sys/4614real ms. Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 51112virt/44596res/6672shr/38068data kb, in 90usr/10sys/101real ms. Pass 3: translated to C into "/tmp/staptXjYUH/stap_18900_src.c" using 51112virt/44596res/6672shr/38068data kb, in 40usr/0sys/36real ms. warning: the compiler differs from the one used to build the kernel The kernel was built by: arm-poky-linux-gnueabi-gcc (GCC) 11.2.0 You are using: gcc (GCC) 11.2.0 Pass 4: compiled C into "stap_18900.ko" in 447970usr/62580sys/174582real ms. Pass 5: starting run. ERROR: Couldn't insert module '/tmp/staptXjYUH/stap_18900.ko': Operation not permitted WARNING: /usr/bin/staprun exited with status: 1 Pass 5: run completed in 120usr/5020sys/5221real ms. Pass 5: run failed. [man error::pass5] 92407 13 alexandre.belloni 2022-01-26 23:47:47 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3522/steps/13/logs/stdio qemuarm-alt ubuntu2004-ty-1 92437 14 alexandre.belloni 2022-01-31 21:45:15 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3553/steps/13/logs/stdio 92438 15 ross.burton 2022-02-01 11:43:54 +0000 That was without my latest changes ???? 92451 16 alexandre.belloni 2022-02-02 22:09:08 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/3990/steps/16/logs/stdio Note that this is not crashing anymore ut the test fails. We can open a new bug if you prefer: AssertionError: 255 != 0 : stap -v --disable-cache -DSTP_NO_VERREL_CHECK -s1 -e 'probe oneshot { print("Hello, "); println("world!") }' WARNING: Kernel function symbol table missing [man warning::symbols] Pass 1: parsed user script and 462 library scripts using 84212virt/74328res/8168shr/65920data kb, in 350usr/50sys/742real ms. Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 85664virt/75908res/8376shr/67372data kb, in 60usr/20sys/80real ms. Pass 3: translated to C into "/tmp/stapPeiRx9/stap_18857_src.c" using 85664virt/75908res/8376shr/67372data kb, in 0usr/0sys/2real ms. warning: the compiler differs from the one used to build the kernel The kernel was built by: aarch64-poky-linux-gcc (GCC) 11.2.0 You are using: gcc (GCC) 11.2.0 92523 17 randy.macleod 2022-02-10 15:54:39 +0000 Seems to be resolved by: https://git.yoctoproject.org/poky/commit/?id=80bddd7c36412f758f1d43036c3f4845cbc87b6c