Created attachment 5158 [details] defconfig used in both yocto compilation and manual compilation Hello, I am reaching out about a problem we are having with our busybox compiled with yocto. I would really appreciate it if you can guide me to the right place to look (or the right place the ask in case this is not related to here). We use a very simple initramfs(contains only busybox) and we give rdinit=/bin/sh to drop to shell in linux kernel 6.1 for a RISC-V cpu. Shell stops responding to basic commands (ls, mkdir exc. ) after some time. The commandline responds to characters though (i can see that it puts the characters to serial line when i type from the keyboard). Here is an example output: [ 5.543467] debug_vm_pgtable: [debug_vm_pgtable ]: Validating architecture page table helpers [ 5.599669] clk: Disabling unused clocks [ 7.981941] Freeing initrd memory: 1668K [ 8.070011] Freeing unused kernel image (initmem) memory: 2112K [ 8.141365] Checked W+X mappings: passed, no W+X pages found [ 8.143958] rodata_test: all tests were successful [ 8.145928] Run /bin/sh as init process /bin/sh: can't access tty; job control turned off ~ # export PATH=/sbin:/bin:/usr/sbin:/usr/bin ~ # ls bin etc lib root usr dev init linuxrc sbin var ~ # cd /bin /bin # ls ls it is not responding right now even though serial works ... (by the way the problem is not specific to ls or the bin directory the above is just an example it happens with mkdir, mount exc. too and within different directories too in ls) What could cause this kind of behaviour? I checked if the kernel or cpu is stuck but they seem to be working fine. There are context-switch'es happening even in this state but the programs doesn't finish (ls, mkdir exc.). The reason i am suspecting the busybox compilation is that i have a manually compiled busybox which works just fine with the same kernel, the confusion for me is that i am using the same defconfig in the problematic busybox that is compiled with yocto as well (i checked the .config files of both compilations and they are same beside the date). I attached the defconfig, OEMAKE flags and do let me know if there is anything else. I am attached to kernel via gdb too if thats needed. EXTRA_OEMAKE="CC='riscv64-poky-linux-gcc --sysroot=/mnt/new-disk/build/tmp/work/riscv64-poky-linux/busybox/1.36.0-r0/recipe-sysroot' LD='riscv64-poky-linux-gcc --sysroot=/mnt/new-disk/build/tmp/work/riscv64-poky-linux/busybox/1.36.0-r0/recipe-sysroot' V=1 ARCH=riscv64 CROSS_COMPILE=riscv64-poky-linux- SKIP_STRIP=y HOSTCC='gcc ' HOSTCPP='gcc -E'" Thank you for your time. Regards, Osman
Trevor has some questions...
Can you provide some more info on the setup and behavior: 1. Are you running this in QEMU, or on real hardware? If the latter, which hardware? 2. Is this deterministic in any way? Does it just happen after running a certain set of commands in specific directories? For example, if you boot the system and do nothing, does it become unresponsive after 5-10 minutes even without input, or do you have to run commands to trigger the issue?
(In reply to Trevor Gamblin from comment #2) > Can you provide some more info on the setup and behavior: > > 1. Are you running this in QEMU, or on real hardware? If the latter, which > hardware? > 2. Is this deterministic in any way? Does it just happen after running a > certain set of commands in specific directories? For example, if you boot > the system and do nothing, does it become unresponsive after 5-10 minutes > even without input, or do you have to run commands to trigger the issue? 1 - This runs on a CPU that is emulated via FPGA, its single core so no SMP. 2 - In my experience, its not bound to a directory but it could be bound to filesystem commands (i used ls, mkdir, mount(only on the virtual filesystems like procfs and sysfs that is), its not bound to a spesific directory from what i observed (for example i had examples where the shell got stuck when i enter ls in both /bin directory and / directory). I checked if it was related to time too but it doesn't seem like it, cause the shell stalls after i type the command even after waiting. Couple of side notes if i forgot to mention: the shell is an initramfs and the shell is given with this in cmdline "rdinit=/bin/sh" so it should not be related to any disk related fault, another thing is that i see the processes that are being created from gdb when i type the command (i see ls, mkdir exc.) the difference between the stalling example is that the created processes do not finish. I checked the struct task_struct of the 'stalled' processes and their fields are like this: __state = 0x2001, preempt_count = 0x2, So the tasks are interruptible. I would be happy to provide more info, i am just not sure where search the fault this kind of problem. Regards
Trevor, It was agreed at the Triage meeting you would update this bug.
Hi Osman, This is unfortunately going to be a hard one for us to triage effectively because it's on a platform (FPGA-emulated) that we cannot easily reproduce ourselves or be sure of full functionality for. We have seen some intermittent issues with riscv64 performance in QEMU, but that is unlikely to map to your situation in a way that we can speak to with confidence. Have you tried building the same image in qemuriscv64 and trying to reproduce the issue?
Need more info to make a judgment on whether this is a real problem.