| Summary: | segfault in matchbox-windowmanager on qemux86-64 intermittently | ||
|---|---|---|---|
| Product: | [Yocto Project Subprojects] Matchbox | Reporter: | Richard Purdie <richard.purdie> |
| Component: | matchbox | Assignee: | Ross Burton <ross.burton> |
| Status: | ACCEPTED --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | alex.kanavin, alexandre.belloni, anuj.mittal, mathieu.dubois-briand, mingli.yu, randy.macleod, steve, yoann.congal |
| Version: | unspecified | ||
| Target Milestone: | 6.2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Richard Purdie
2020-05-20 03:43:00 UTC
In 165 tests, this only happened once but three have been other cases in the YP AB. Add debug code in case it happens again. Failure from qa-extras2 step 5c this time: https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1963 Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/runtime/cases/parselogs.py", line 375, in test_parselogs
self.assertEqual(errcount, 0, msg=self.msg)
AssertionError: 1 != 0 : Log: /home/pokybuild/yocto-worker/qa-extras2/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/target_logs/dmesg_output.log
-----------------------
Central error: [ 8.089153] matchbox-window[231]: segfault at 8 ip 00007f6e2896073c sp 00007ffef993a0a0 error 4 in libstartup-notification-1.so.0.0.0[7f6e28960000+4000]
***********************
[ 5.692134] systemd[1]: Started udev Kernel Device Manager.
[ 5.710011] systemd[1]: Finished udev Coldplug all Devices.
[ 5.740845] systemd[1]: Finished Load/Save Random Seed.
[ 6.220586] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
[ 6.230456] systemd[1]: Finished Rebuild Dynamic Linker Cache.
[ 6.241901] systemd[1]: Started Journal Service.
[ 6.254118] systemd-journald[137]: Received client request to flush runtime journal.
[ 7.997486] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 7.999978] NFSD: Using legacy client tracking operations.
[ 8.005390] NFSD: starting 90-second grace period (net f0000098)
[ 8.089153] matchbox-window[231]: segfault at 8 ip 00007f6e2896073c sp 00007ffef993a0a0 error 4 in libstartup-notification-1.so.0.0.0[7f6e28960000+4000]
[ 8.089164] Code: ee 48 89 df e8 e5 fa ff ff 48 83 c5 01 41 39 6c 24 50 49 89 45 00 7f de 49 8b 7c 24 10 44 89 fe 31 d2 e8 47 fc ff ff 48 89 c7 <8b> 40 08 41 89 44 24 20 e8 17 f9 ff ff 49 8b 7c 24 10 44 89 f6 31
***********************
1 errors found in logs.
Stdout:
Machine information:
*******************************
Machine name: qemux86-64
CPU: Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
Arch: x86_64
Physical cores: 1
Logical cores: 1
*******************************
startup-notification hasn't seen any commits since 2014: https://cgit.freedesktop.org/startup-notification/log/ and is implementing notifications over X protocol between X clients. Which begs the question: should we take the easy way out and just disable it in oe-core, specifically matchbox? It's all obsolete technology. startup-notification is only obsolete if you consider X obsolete. The protocol is 'done' but it's a key part of X desktops, its just that mutter/kde don't use libstartupnotification directly. Seems like the error only occurs when systemd is in use? So far all error logs I have seen are either qa-extras2 steps 5/7 or altcfg builds.
I have read through the code, and:
a) startup notification does not handle errors, and instead proceed directly to accessing a struct member ---> crash (boo!):
atom_reply = xcb_intern_atom_reply(display->xconnection,
atom_utf8_string_c,
NULL);
display->UTF8_STRING = atom_reply->atom;
free(atom_reply);
(atom_reply can be NULL!)
b) the only way the error can happen where it does is if the X server can't allocate memory. I suspect that because systemd starts everything at once, there is a brief situation where memory is exhausted.
How about bumping RAM again for core-image-sato, and observing for a while if the problem wont appear anymore?
Patch to aid debugging: https://lists.openembedded.org/g/openembedded-core/message/140067 Alex and I both tried some debugging of this over the weekend. Alex ran 10 builds with 512MB ram and 10 builds with 1024MB whilst there was a master-next build running on the autobuilder. We saw no failures in either set. I also tried running 6 qemu images (mixtures of core-image-sato and core-image-sato-sdk) booted locally, either unbound, or all pinned to a single cpu core whilst a world build of OE-Core was in progress, including a webkit build. No reproduction of the failure either (I have seen it locally once). Merged http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=27f59b5138d734aa466d7465f9438cd93e9ba861 on the offchance it might help. Debug patch from Alex has been in master-next for a while but has never reproduced with that present. Closing as this has stopped happening. Might be a weird 32/64 bit issue in time_t, might have been a bug in another piece of the stack that has been since fixed. Will reopen if happens again. Happening again. *** Bug 14647 has been marked as a duplicate of this bug. *** Ross, Moar logging? Moved en masse from 4.2 to 4.3 by Randy. dunfell https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/7103/steps/14/logs/stdio qemux86-64-alt fedora38-ty-2 Have we seen this recently? qemux86-64-alt ubuntu2204-vk-2 mathieu/master-next completed at 2025-02-04T08:31:17Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/912/steps/13/logs/stdio *** Bug 15741 has been marked as a duplicate of this bug. *** qemux86-alt opensuse155-vk-1 mathieu/master-next completed at 2025-02-13T10:31:34Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/978/steps/14/logs/stdio qa-extras2 debian12-vk-3 master completed at 2025-04-04T02:19:47Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/1314/steps/23/logs/stdio qemux86-64-alt debian11-vk-2 master-next completed at 2025-05-14T16:27:29Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/1586/steps/14/logs/stdio Two failures so close in time: this seems a bit strange, maybe these are related to the content of the master-next branch. qemux86-64-alt debian12-vk-9 master-next completed at 2025-05-14T21:13:39Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/1589/steps/14/logs/stdio qemux86-alt opensuse156-vk-1 master-next completed at 2025-06-02T08:01:48Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/1716/steps/15/logs/stdio qemux86-alt debian12-vk-5 mathieu/master-next completed at 2025-08-27 08:46:59+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/2254/steps/13/logs/stdio qemux86-alt ubuntu2504-vk-1 mathieu/master-next completed at 2025-09-29 17:14:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/2447/steps/14/logs/stdio qa-extras2 ubuntu2404-vk-2 master-next completed at 2025-11-03 12:37:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/2654/steps/24/logs/stdio qa-extras2 debian12-vk-4 master completed at 2025-12-26 01:34:28+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/2914/steps/30/logs/stdio qemux86-alt debian12-vk-4 mathieu/master-next completed at 2025-12-26 17:01:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/2901/steps/14/logs/stdio qa-extras2 debian12-vk-5 stable/2.16-nut&stable/whinlatter-nut&whinlatter-next completed at 2026-01-08 07:02:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/2973/steps/24/logs/stdio qemux86-64-alt rocky9-vk-2 mathieu/master-next completed at 2026-01-08 18:34:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/2957/steps/14/logs/stdio Definitely notable that the failures are all in -altcfg builds (systemd) or systemd-using builds in qa-extras. Some hunches: There's long-standing time_t problems in libstartup-notification where it's possible that some client code is using a 32-bit int and chaos follows. See: - https://gitlab.freedesktop.org/xdg/startup-notification/-/merge_requests/1 - https://gitlab.freedesktop.org/xdg/startup-notification/-/issues/6 That doesn't explain why it's only seen on systemd builds. I wondered if the recent race/dependency fix (https://lore.kernel.org/openembedded-core/20251129040503.3958950-3-Qi.Chen@windriver.com/) was related, but we've had reports of this problem after that merged so that isn't the fix. I do wonder if the problem is that the X server restarts on some boots and the clients are no handling that gracefully. Obviously in this case, stopping it restarting is the answer. qa-extras2 fedora42-vk-1 mathieu/master-next completed at 2026-03-10 16:39:19+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/3310/steps/24/logs/stdio qemux86-64-alt ubuntu2404-vk-2 mathieu/master-next completed at 2026-04-07 15:22:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/3503/steps/14/logs/stdio qemux86-64-alt ubuntu2404-vk-3 mathieu/master-next completed at 2026-04-09 12:11:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/3514/steps/14/logs/stdio qemux86-alt ubuntu2404-vk-3 master completed at 2026-07-12 01:16:25+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4069/steps/15/logs/stdio Potentially interesting/related: matchbox-panel crashing in libstartup-notification. https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4069 qa-extras2 debian13-vk-1 master-next completed at 2026-07-21 23:48:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/4146/steps/24/logs/stdio qemux86-alt rocky9-vk-2 mathieu/master-next completed at 2026-07-23 03:10:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4134/steps/15/logs/stdio I have a lead, one of Ross' theories is looking correct and I have evidence. Whilst https://valkyrie.yocto.io/pub/non-release/20260726-112/artefacts/qemux86-alt/oeqa-target-artefacts-20260726-0kkx0pfa/tests_artifacts.tar.gz doesn't contain any coredump/stacktrace, It does contain both an Xorg.log and an Xorg.log.old. Xorg.log.old spans 4.116 -> 4.561s and ends with: [ 4.559] (II) UnloadModule: "libinput" [ 4.560] (II) UnloadModule: "libinput" [ 4.561] (WW) xf86CloseConsole: KDSETMODE failed: Input/output error [ 4.561] (WW) xf86CloseConsole: VT_GETMODE failed: Input/output error [ 4.561] (WW) xf86OpenConsole: VT_GETSTATE failed: Input/output error [ 4.561] (II) Server terminated successfully (0). Closing log file. and then Xorg.log starts at 4.947s. Looksing at the failed build: https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4163 8[ 4.790969] matchbox-panel[355]: segfault at 8 ip b56237d2 sp bfc46880 error 4 in libstartup-notification-1.so.0.0.0[37d2,b5623000+5000] likely on CPU 1 (core 1, socket 0) [ 4.797202] Code: 08 55 83 c5 01 ff 74 24 10 e8 0a fa ff ff 83 c4 10 39 6e 30 89 07 7f e0 8b 6c 24 10 83 ec 04 6a 00 55 ff 76 08 e8 9e fb ff ff <8b> 50 08 89 56 10 89 04 24 e8 10 f9 ff ff 83 c4 0c 6a 00 ff 74 24 i.e. the crash is neatly between xserver restarts. qemux86-alt ubuntu2510-vk-1 master&master-next completed at 2026-07-26 18:45:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4163/steps/15/logs/stdio qa-extras2 debian12-vk-6 wrynose completed at 2026-07-27 13:16:29+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/4188 qemux86-alt debian12-vk-7 master&master-next completed at 2026-08-18 13:41:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4359/steps/17/logs/stdio qemux86-alt debian12-vk-7 mathieu/master-next completed at 2026-08-20 18:41:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4383/steps/16/logs/stdio qemux86-alt ubuntu2204-vk-1 master-next&master completed at 2026-08-30 08:53:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4466/steps/17/logs/stdio qemux86-alt debian12-vk-5 master completed at 2026-09-25 02:05:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/4647/steps/17/logs/stdio qa-extras2 debian13-vk-2 mathieu/master-next completed at 2026-10-18 14:09:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/26/builds/4808/steps/26/logs/stdio It looks like xserver typically restarts during boot, I suspect the crash is when some bits of the X session have started already and don't handle the xserver disappearing. Unsetting SPLASH, to remove psplash, appears to remove the restart. This suggests a problem with the handoff from psplash to xserver. |