Bug 15100

Summary: qemuarm: core-image-sato UI hangs with 6.1 kernel
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Randy MacLeod <randy.macleod>
Component: kernelAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: narpat.mali, tom.zanussi
Version: 4.2   
Target Milestone: 4.3 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
poky-master-qemuarm--core-image-sato-frozen none

Description Randy MacLeod 2023-04-11 22:06:16 UTC
Created attachment 4947 [details]
poky-master-qemuarm--core-image-sato-frozen

When Narpat was testing chromium in qemuarm (and qemuarm64), he noted that for poky/master the UI  would come up but be 'frozen and greyed out'.  

to reproduce, use poky/master and:
$ bitbake core-image-sato
$ runqemu qemuarm serialstdio publicvnc qemuparams="-m 512"
$ run your vncviewer
See attached.

I confirmed this to be the case and did a qemuarm git bisect to find that the problematic commit is apparently:

❯ git bisect good                                                                                         
b8cac140f4ec37959b788261a0d5559497ce9c1b is the first bad commit                           
commit b8cac140f4ec37959b788261a0d5559497ce9c1b                                                          
Author: Bruce Ashfield <bruce.ashfield@gmail.com>                                                        
Date:   Wed Mar 1 10:13:43 2023 -0500                                                                    
                                                                                                         
    linux-yocto/6.1: update to v6.1.14                                                                   
                                                                                                         
    Updating  to the latest korg -stable release that comprises                                       
    the following commits:       
  ...

FYI, here's the log:
❯ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [71c316576150022a2337777da8274318064ef73f] Revert "ripgrep: add ripgrep-13.0.0"
git bisect bad 71c316576150022a2337777da8274318064ef73f
# status: waiting for good commit(s), bad commit known
# bad: [1ace5646156640ceb9322dd8cb322a58772936a5] puzzles: upgrade to latest revision
git bisect bad 1ace5646156640ceb9322dd8cb322a58772936a5
# status: waiting for good commit(s), bad commit known
# good: [dfa8b896c64182fbf7c22eef4ae7e1948f5279c4] webkitgtk: Fix build on 32bit arm
git bisect good dfa8b896c64182fbf7c22eef4ae7e1948f5279c4
# good: [35b187374905d4e3f0e6a66b183304b56aebe948] libxpm: update 3.5.13 -> 3.5.14
git bisect good 35b187374905d4e3f0e6a66b183304b56aebe948
# good: [557aab215884d10839fa24f797890c99f38e97b2] bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location
git bisect good 557aab215884d10839fa24f797890c99f38e97b2
# good: [069e5df6ae3b23659c91a59fd6990fc1a8fd9a92] pkgconfig: use system glib for nativesdk builds
git bisect good 069e5df6ae3b23659c91a59fd6990fc1a8fd9a92
# good: [7a548930d27af2b3034fabe18f51a695d121c70c] musl: Update to tip of trunk
git bisect good 7a548930d27af2b3034fabe18f51a695d121c70c
# bad: [44913cfab42856b8f40e13bac76c5abfa1df0cb2] linux: inherit pkgconfig in kernel.bbclass
git bisect bad 44913cfab42856b8f40e13bac76c5abfa1df0cb2
# good: [33d9b3a8ed186c2d212acb83f715367912010e7f] systemd-systemctl: Create machine-id with "uninitialized" text in it
git bisect good 33d9b3a8ed186c2d212acb83f715367912010e7f
# good: [afd1545a5cead40219727b427607c44e6c192504] linux-yocto/6.1: update to v6.1.12
git bisect good afd1545a5cead40219727b427607c44e6c192504
# bad: [d24ae96060c83604c1eb5a4a2ca3503f827e9bfb] vim: add missing pkgconfig inherit
git bisect bad d24ae96060c83604c1eb5a4a2ca3503f827e9bfb
# bad: [64af4e8abbac727a42ab685471129d1ffbe99b4f] linux-yocto/5.15: update to v5.15.96
git bisect bad 64af4e8abbac727a42ab685471129d1ffbe99b4f
# bad: [b8cac140f4ec37959b788261a0d5559497ce9c1b] linux-yocto/6.1: update to v6.1.14
git bisect bad b8cac140f4ec37959b788261a0d5559497ce9c1b
# good: [aedc70ea0e3df900f04b409979a1dcd99930ab21] linux-yocto/5.15: update to v5.15.94
git bisect good aedc70ea0e3df900f04b409979a1dcd99930ab21
# first bad commit: [b8cac140f4ec37959b788261a0d5559497ce9c1b] linux-yocto/6.1: update to v6.1.14


I haven't yet tried to trace init to see what syscall(s) results in the UI hanging.
Comment 1 Randy MacLeod 2023-04-13 14:39:01 UTC
Randy and Narpat to debug userspace to narrow down the issue.
Comment 2 Narpat Mali 2023-04-18 18:04:48 UTC
Tested with linux-yocto-dev 6.3.0-rc6-yoctodev-standard+ and the hang doesn't happen. This seems like a kernel regression.

Using poky-master: ff633ce7a7ffc0d39b86043caeb63a9ffcef99b8

and include below in conf/local.conf
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"

Next step is to compare the boot logs between 6.1.20 & 6.3.0 and sato initialization to understand the how the deadlock is occurring.
Comment 3 Randy MacLeod 2023-04-20 16:30:57 UTC
FYI, sato UI freeze also seen in poky/master =

f79046d082 (origin/master, origin/HEAD) cve-exclusions: Document some further linux-yocto CVE statuses

for qemuriscv64.
Comment 4 Ross Burton 2023-04-20 16:46:30 UTC
Using the head of mickledore branch and qemuarm, I don't see this. I do have a slew of tweaks in my local.conf so I'll do another build with a clean conf.
Comment 5 Ross Burton 2023-04-20 18:35:20 UTC
OK I can replicate with sysv.  Curious!
Comment 6 Randy MacLeod 2023-04-20 18:43:52 UTC
Yes, all my testing was with sysvinit. I should try xfce with sysvinit since the default for WR Linux is systemd.

For completeness, I confirmed that for qemuriscv64, switching to the 6.3 kernel using:
 
   PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"

also avoids the sato UI hang.

I also switched to mickeldore:
4bb775aecb (HEAD -> mickledore, origin/mickledore) 
           cve-exclusions: Document some further linux-yocto CVE statuses

and back to the standard kernel and the problem exist there fore me as well. 

It's odd that you can't reproduce it since both Narpat and I do see the problem and I even saw it when building on my RPi4 running Wind River Linux and qemu as a guest so it's not just a host issue. For builds on intel, I connected using runqemu publicvnc but when testing on the RPi4/WRL it's local graphics so that takes the rendering out of the picture as a factor.
Comment 7 Ross Burton 2023-04-20 18:48:16 UTC
So a potential lead is that when it works the VNC connection is 32bits-per-pixel rgb888, but when it doesn't the connection is 8bpp rgb222.  Which doesn't sounds right _at all_.
Comment 8 Ross Burton 2023-04-21 13:52:59 UTC
Found the bug in the kernel, and found the fix.  Typically, it was broken from 6.1.13 to 6.1.21, and we were trying to release with 6.1.20.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.1.y&id=1ea3e18e53f2e741b264960a7f829f44110d72d2 is the fix.  I suggest we pick that into linux-yocto and let it rebase out when we upgrade past 6.1.20.
Comment 9 Randy MacLeod 2023-04-21 14:01:45 UTC
How did you find the bug/fix?
Comment 10 Ross Burton 2023-04-21 14:16:19 UTC
Verified that 6.1.14 broke but 6.1.12 worked. Bisected that further to 6.1.13 is broken too. That reduced it down to a manageable number of commits which I bisected.

Thanks to good commit messages in the kernel it was trivial to find the commit in master which fixed the buggy commit.
Comment 11 Randy MacLeod 2023-04-21 15:49:03 UTC
Thanks. That's disappointingly banal! ;-)
Comment 12 Ross Burton 2023-04-26 15:24:19 UTC
Fixed with the upgrade to kernel 6.1.25 in f3a422f02113cda2aa7872e350e468bdb447c8c3.