Created attachment 4947 [details] poky-master-qemuarm--core-image-sato-frozen When Narpat was testing chromium in qemuarm (and qemuarm64), he noted that for poky/master the UI would come up but be 'frozen and greyed out'. to reproduce, use poky/master and: $ bitbake core-image-sato $ runqemu qemuarm serialstdio publicvnc qemuparams="-m 512" $ run your vncviewer See attached. I confirmed this to be the case and did a qemuarm git bisect to find that the problematic commit is apparently: ❯ git bisect good b8cac140f4ec37959b788261a0d5559497ce9c1b is the first bad commit commit b8cac140f4ec37959b788261a0d5559497ce9c1b Author: Bruce Ashfield <bruce.ashfield@gmail.com> Date: Wed Mar 1 10:13:43 2023 -0500 linux-yocto/6.1: update to v6.1.14 Updating to the latest korg -stable release that comprises the following commits: ... FYI, here's the log: ❯ git bisect log git bisect start # status: waiting for both good and bad commits # bad: [71c316576150022a2337777da8274318064ef73f] Revert "ripgrep: add ripgrep-13.0.0" git bisect bad 71c316576150022a2337777da8274318064ef73f # status: waiting for good commit(s), bad commit known # bad: [1ace5646156640ceb9322dd8cb322a58772936a5] puzzles: upgrade to latest revision git bisect bad 1ace5646156640ceb9322dd8cb322a58772936a5 # status: waiting for good commit(s), bad commit known # good: [dfa8b896c64182fbf7c22eef4ae7e1948f5279c4] webkitgtk: Fix build on 32bit arm git bisect good dfa8b896c64182fbf7c22eef4ae7e1948f5279c4 # good: [35b187374905d4e3f0e6a66b183304b56aebe948] libxpm: update 3.5.13 -> 3.5.14 git bisect good 35b187374905d4e3f0e6a66b183304b56aebe948 # good: [557aab215884d10839fa24f797890c99f38e97b2] bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location git bisect good 557aab215884d10839fa24f797890c99f38e97b2 # good: [069e5df6ae3b23659c91a59fd6990fc1a8fd9a92] pkgconfig: use system glib for nativesdk builds git bisect good 069e5df6ae3b23659c91a59fd6990fc1a8fd9a92 # good: [7a548930d27af2b3034fabe18f51a695d121c70c] musl: Update to tip of trunk git bisect good 7a548930d27af2b3034fabe18f51a695d121c70c # bad: [44913cfab42856b8f40e13bac76c5abfa1df0cb2] linux: inherit pkgconfig in kernel.bbclass git bisect bad 44913cfab42856b8f40e13bac76c5abfa1df0cb2 # good: [33d9b3a8ed186c2d212acb83f715367912010e7f] systemd-systemctl: Create machine-id with "uninitialized" text in it git bisect good 33d9b3a8ed186c2d212acb83f715367912010e7f # good: [afd1545a5cead40219727b427607c44e6c192504] linux-yocto/6.1: update to v6.1.12 git bisect good afd1545a5cead40219727b427607c44e6c192504 # bad: [d24ae96060c83604c1eb5a4a2ca3503f827e9bfb] vim: add missing pkgconfig inherit git bisect bad d24ae96060c83604c1eb5a4a2ca3503f827e9bfb # bad: [64af4e8abbac727a42ab685471129d1ffbe99b4f] linux-yocto/5.15: update to v5.15.96 git bisect bad 64af4e8abbac727a42ab685471129d1ffbe99b4f # bad: [b8cac140f4ec37959b788261a0d5559497ce9c1b] linux-yocto/6.1: update to v6.1.14 git bisect bad b8cac140f4ec37959b788261a0d5559497ce9c1b # good: [aedc70ea0e3df900f04b409979a1dcd99930ab21] linux-yocto/5.15: update to v5.15.94 git bisect good aedc70ea0e3df900f04b409979a1dcd99930ab21 # first bad commit: [b8cac140f4ec37959b788261a0d5559497ce9c1b] linux-yocto/6.1: update to v6.1.14 I haven't yet tried to trace init to see what syscall(s) results in the UI hanging.
Randy and Narpat to debug userspace to narrow down the issue.
Tested with linux-yocto-dev 6.3.0-rc6-yoctodev-standard+ and the hang doesn't happen. This seems like a kernel regression. Using poky-master: ff633ce7a7ffc0d39b86043caeb63a9ffcef99b8 and include below in conf/local.conf PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev" Next step is to compare the boot logs between 6.1.20 & 6.3.0 and sato initialization to understand the how the deadlock is occurring.
FYI, sato UI freeze also seen in poky/master = f79046d082 (origin/master, origin/HEAD) cve-exclusions: Document some further linux-yocto CVE statuses for qemuriscv64.
Using the head of mickledore branch and qemuarm, I don't see this. I do have a slew of tweaks in my local.conf so I'll do another build with a clean conf.
OK I can replicate with sysv. Curious!
Yes, all my testing was with sysvinit. I should try xfce with sysvinit since the default for WR Linux is systemd. For completeness, I confirmed that for qemuriscv64, switching to the 6.3 kernel using: PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev" also avoids the sato UI hang. I also switched to mickeldore: 4bb775aecb (HEAD -> mickledore, origin/mickledore) cve-exclusions: Document some further linux-yocto CVE statuses and back to the standard kernel and the problem exist there fore me as well. It's odd that you can't reproduce it since both Narpat and I do see the problem and I even saw it when building on my RPi4 running Wind River Linux and qemu as a guest so it's not just a host issue. For builds on intel, I connected using runqemu publicvnc but when testing on the RPi4/WRL it's local graphics so that takes the rendering out of the picture as a factor.
So a potential lead is that when it works the VNC connection is 32bits-per-pixel rgb888, but when it doesn't the connection is 8bpp rgb222. Which doesn't sounds right _at all_.
Found the bug in the kernel, and found the fix. Typically, it was broken from 6.1.13 to 6.1.21, and we were trying to release with 6.1.20. https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.1.y&id=1ea3e18e53f2e741b264960a7f829f44110d72d2 is the fix. I suggest we pick that into linux-yocto and let it rebase out when we upgrade past 6.1.20.
How did you find the bug/fix?
Verified that 6.1.14 broke but 6.1.12 worked. Bisected that further to 6.1.13 is broken too. That reduced it down to a manageable number of commits which I bisected. Thanks to good commit messages in the kernel it was trivial to find the commit in master which fixed the buggy commit.
Thanks. That's disappointingly banal! ;-)
Fixed with the upgrade to kernel 6.1.25 in f3a422f02113cda2aa7872e350e468bdb447c8c3.