For debugging purposes, it would be nice to record the host dmesg at the start of bitbake and the end of the build to be able to do a diff. This would probably uncover interesting failures that are currently hidden/ignored by the underlying build system. We know there are gcc/ld segfaults and sometime qemu segfaults. It would be nice to have this in a class that could be then optionally inherited to enable the feature. There are bb.event.BuildStarted and bb.event.BuildCompleted events to listen to.
We've enabled unprivileged user to run dmesg for the Yocto Autobuilder. We still need to make this optional since not all users will be able to run dmesg.
Christos send out some design ideas and Richard reviewed them.
hi just a kind reminder that I have sent a patch one week ago. https://lists.openembedded.org/g/openembedded-core/topic/patch_host_dmesg_logger/113571912 As soon as it is merged in 'master' I will send a patch for the documentation. Br Christos
I just pinged Mathieu on IRC since it seems that he missed your patch. It's here: https://patchwork.yoctoproject.org/project/oe-core/patch/20250610153125.4858-1-gavrosc@yahoo.com/ but not here: https://git.yoctoproject.org/poky/log/?h=master-next&qt=grep&q=host_dmesg_logger I'll also reply to the email with minor comments.
Christos, I remembered a rust build issue that we've seen in dmesg for a while. The signature is: [Thu Jun 12 05:20:53 2025] rustc[2697328]: segfault at 7f0f989d2ff8 ip 00007f0f9fffe955 sp 00007f0f989d3000 error 6 in librustc_driver-0b4e066e202f8ce4.so[7f0f9a34e000+6682000] where I'm running dmesg -T to get a human, readable timestamp. Do you see this is your testing if you build rust? Do you think we should have dmesg to use -T ?
Hi Randy, i have built git and bash during my tests. The build of bash gave as diff the following during one of the tests: ================================= WARNING: === dmesg diff === WARNING: --- dmesg_start WARNING: +++ dmesg_end WARNING: @@ -1176,3 +1176,131 @@ WARNING: [ 6754.903425] intel_powerclamp: Stop forced idle injection WARNING: [ 6755.912747] intel_powerclamp: Start idle injection to reduce power WARNING: [ 6788.303522] intel_powerclamp: Stop forced idle injection WARNING: +[ 6849.524870] intel_powerclamp: Start idle injection to reduce power WARNING: +[ 6853.567757] intel_powerclamp: Stop forced idle injection WARNING: +[ 6854.577562] intel_powerclamp: Start idle injection to reduce power WARNING: +[ 6855.594734] intel_powerclamp: Stop forced idle injection WARNING: +[ 6856.607049] intel_powerclamp: Start idle injection to reduce power WARNING: +[ 6928.472768] intel_powerclamp: Stop forced idle injection WARNING: +[ 6933.501532] intel_powerclamp: Start idle injection to reduce power WARNING: +[ 6967.920338] intel_powerclamp: Stop forced idle injection WARNING: +[ 6968.931633] intel_powerclamp: Start idle injection to reduce power WARNING: +[ 6972.531740] NOHZ tick-stop error: local softirq work is pending, handler #200!!! WARNING: +[ 6981.307765] NOHZ tick-stop error: local softirq work is pending, handler #200!!! WARNING: +[ 6981.691663] NOHZ tick-stop error: local softirq work is pending, handler #200!!! WARNING: +[ 6993.683581] NOHZ tick-stop error: local softirq work is pending, handler #03!!! WARNING: +[ 6996.543603] NOHZ tick-stop error: local softirq work is pending, handler #200!!! WARNING: +[ 6997.183555] NOHZ tick-stop error: local softirq work is pending, handler #10!!! WARNING: +[ 6998.103532] NOHZ tick-stop error: local softirq work is pending, handler #200!!! WARNING: +[ 7009.253522] NOHZ tick-stop error: local softirq work is pending, handler #200!!! .. .. .. ================================= I will try rust and the option -T and come back to you. I read your comments in the patch. I will wait if someone comments something more and i will prepare a v2. Br Christos
Created attachment 5123 [details] recipes_results_logger_20250623 diff logger for 9 recipes
hi i have attached the results of building 9 recipes(recipes_results_logger_20250623.zip). I have not seen the same issue for rust. i can include '-T' option in v2. i guess it looks more user friendly. Br Christos
Thanks for the additional info Christos. It's a shame that your system is so "noisy" due to the intel_powerclamp and the "NOHZ tick-stop error" logs since the expected output of this dmesg diff tool for most builds is NOTHING. I'll try v2 on a less noisy server.
Richard would like to get Antonin to test this on YP AB once we agree on v2. Randy to message him on IRC.
hi I have sent out a v2: https://lists.openembedded.org/g/openembedded-core/topic/patch_v2_host_dmesg_logger/113792825
I believe in initial discussions over IRC, I'd suggested there would need to be a somewhat dynamic "ignore" list to feed the diff into "grep -v -f ignore.txt". This would be how to deal with the powerclamp and NOHZ examples here, so that they are only a one-shot distraction, and not annoying people time and time again.
Hi Paul, thank you for your comment. Richard ran 2 builds using the patch2 in AB. There is a discussion here if we have to use filters or levels. https://lists.openembedded.org/g/openembedded-core/topic/113792825#msg219300 Is the solution you're proposing consistent with the rest of Yocto? I was under the impression that in most of these cases, conditions like these are hardcoded. Br Christos