Hi Bruce,

Do you know if there's any way to disable GCC_PLUGINS in linux-yocto kernel?
I tried setting CONFIG_HAVE_GCC_PLUGINS=n, but it does not work.

.kernel-meta/cfg/merge_config_build.log:Value requested for CONFIG_HAVE_GCC_PLUGINS not in final .config
.kernel-meta/cfg/merge_config_build.log:Requested value:  CONFIG_HAVE_GCC_PLUGINS=n
.kernel-meta/cfg/merge_config_build.log:Actual value:     CONFIG_HAVE_GCC_PLUGINS=y

Do you have any idea how to disable it?

Regards,
Qi

Is this with poky/OE core master ?

We've discussed this on several occasions, and on a few, the issues were specific with WRL, and couldn't be reproduced with OE.

We don't want to disable the plugins, but they do have a component of detection. I fix the auto.cmd values during devsrc packaging so they don't cause issues on target.

What is the exact compilation issue that you are seeing ?

I'll run some tests, my plugins fixups may have broken again.

(In reply to comment #2)
> Is this with poky/OE core master ?
> 

Hi Bruce, thanks for your reply.
The problem of different .config between linux-yocto and kernel-devsrc exist in both master and hardknott.
That said, the error I encountered is about building kernel module inside SDK.
I just tried running `make scripts prepare' on target, no such issue.

> We've discussed this on several occasions, and on a few, the issues were
> specific with WRL, and couldn't be reproduced with OE.
> 

Is OE's SDK going to support building out-of-tree kernel modules?
I can vaguely recall that you replied in some email that it's not supported now and eSDK is suggested.

> We don't want to disable the plugins, but they do have a component of
> detection. I fix the auto.cmd values during devsrc packaging so they don't
> cause issues on target.
> 

Could you please tell me what the fix is, where could I find related info?

> What is the exact compilation issue that you are seeing ?
> 

I'm seeing the following error.
  CC      scripts/mod/empty.o
cc1: error: cannot load plugin ./scripts/gcc-plugins/arm_ssp_per_task_plugin.so: ./scripts/gcc-plugins/arm_ssp_per_task_plugin.so: undefined symbol: _ZN8opt_pa
ss14set_pass_paramEjb

I first wanted to look at why this lib is not loadable, but then I found that the linux-yocto kernel's .config file is different from the .config in kernel-devsrc package. The following two configs are enabled in kernel-devsrc but not in linux-yocto's .config.
CONFIG_GCC_PLUGINS=y
CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK=y
My instinct is this would be a problem, so I opened this yocto bug.
Do you think this would be a problem?

Last question is: if I really want to disable the GCC plugins for my kernel locally, how could I do it? I tried using kernel config fragment but it did not work.

> I'll run some tests, my plugins fixups may have broken again.

Hi Bruce,

Below is the full difference between kernel's config and kernel-devsrc's config. It seems that the major concern is still about gcc plugin.

$ diff tmp/work/qemuarm-poky-linux-gnueabi/linux-yocto/5.15.13+gitAUTOINC+9855687d02_aa48e73490-r0/linux-qemuarm-standard-build/.config tmp/work-shared/qemuarm/kernel-build-artifacts/.config
468a469
> CONFIG_STACKPROTECTOR_PER_TASK=y
647a649,652
> CONFIG_GCC_PLUGINS=y
> # CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
> # CONFIG_GCC_PLUGIN_RANDSTRUCT is not set
> CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK=y
3787a3793,3795
> # CONFIG_GCC_PLUGIN_STRUCTLEAK_USER is not set
> # CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF is not set
> # CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL is not set

Hi Bruce,

I've added info in above comments and re-assigned the bug to you.
I realized that after Randy changed this bug's assignee to me, you might not get the email with updated comments.

Regards,
Qi

Thanks for the update! And yes, I had missed this.

I'll have a look at this in more detail shortly.

Thanks for the update! And yes, I had missed this.

I'll have a look at this in more detail shortly.

I haven't forgotten about this, I am tied up on another priority issue with golang, but will get to this ASAP.

Bruce asked us to share some details of problems we have been having building Arm kernels using the SDK compilers since he thinks they are related to the issue reported here. So FWIW here goes...

Our starting point is different to the reporter (can't compile the kernel for ARCH=arm using SDK) but there are definitely features in common.

I've done my best to break this down into smaller facts to help support
the final summary at the bottom.

1. Currently an SDK toolset cannot compile an ARCH=arm multi_v7_defconfig
   kernel. So far I have tested dunfell, honister and master. The SDK was
   generated for qemuarm using core-image-minimal on a clean poky checkout).

   No subtly is needed to reproduce (confirmed to fail using v5.15):

   # Source the SDK environment and change directory into Linux sources
   make mrproper
   make defconfig
   make
   
2. The problem is likely generic: on all kernel versions that support plugins
   there can be uncompilable configurations that (I think) trace back to the
   SDK. However the issue is especially problematic for ARCH=arm kernels
   because arm kernels automatically enable a security related plugin based
   solely on the value of CONFIG_GCC_PLUGINS.

3. CONFIG_GCC_PLUGINS is default y symbol and will be enabled whenever
   possible. However it does have a dependency based on a plugin header
   check (e.g. it is forced off if gcc does not support plugins).

   a) In v5.11+ the check this is based on the simply on existence of
      include/plugin-version.h (which the SDK provides and thus
      GCC_PLUGINS is always enabled). This often leads to compiler errors
      because, although the SDK includes this header, the SDK does not
      include dependencies required to build plugins (such as mpc.h and
      mpfr.h).

   b) In v5.10 and earlier the check does a compilability check rather
      than a simple presence/absence test and this often fails for SDK
      systems:

      i)  If the workstaion does not include libmpc-dev and libmpfr-dev
          then the test fails and GCC_PLUGINS=n
      ii) (For gcc-11) The plugin API requires -std=gnu++11 rather than
          -std=gnu++98 (this is one of the reasons to simplify the tests
	  in v5.11).
	
      It should be noted that if the compilability check (spuriously)
      fail due to missing headers this has the effect of disabling
      CONFIG_GCC_PLUGINS. The kernel can sucessfully be built with
      without the extra features.

   c) Regardless of the kernel version, if CONFIG_GCC_PLUGINS is y,
      then we cannot run the compiler due to symbol errors:
      
      cc1: error: cannot load plugin \
      ./scripts/gcc-plugins/arm_ssp_per_task_plugin.so: \
      ./scripts/gcc-plugins/arm_ssp_per_task_plugin.so: \
      undefined symbol: _ZN8opt_pass14set_pass_paramEjb

4. There is something funny going on with the SDK binaries. Specifically
   cc1 does not appear to contain any symbols from the plugin API.
   Symbols like _ZN8opt_pass14set_pass_paramEjb are found in some of the
   intermediate binaries but do not survice the splitting off of debug
   information.

   I used the following command to help me reach this conclusion:

   for i in `find sdk tmp/work -name "cc1"`
   do
       strings $i | grep -q opt_pass.*set_pass_param \
           && echo "OK: $i" \
	   || echo "FAIL: MISSING PLUGIN API: $i"
   done | tee bad_cc1.log

5. I don't *think* the unstripped binaries work either (which implies
   that the problem isn't the result of the debug info split.

   The reason for the "*think*" is that my testing associated with this
   fact is a bit crazy and could easily have an experimental error
   if I'm missed some detail about how the SDK dynamic linking works:

   cp tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-arm/11.2.0-r0/image/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/libexec/arm-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/11.2.0/cc1 sdk/sysroots/x86_64-pokysdk-linux/usr/libexec/arm-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/11.2.0/cc1
   sudo mkdir -p /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/lib/
   sudo cp cp sdk/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/lib/


In short, I think there could be two problems related to GCC plugins in the current SDKs. Firstly some of the dependent headers needed to consume the GCC plugin header files (mpc.h, mpfr.h) are missing from the SDK. Secondly something has gone wrong in the generation of cc1 and, as a result, the plugin related C++ symbols are missing from the symbol tables in cc1.

This is going to miss m4, but I'm still working on a solution. Once found, we can backport it to the -stable LTS.

I posted a possible fix for this at https://lists.openembedded.org/g/openembedded-core/topic/rfc_build_linux_with_a_oe/90687587. That can likely be improved, so there is no need for guessing arguments.

I can confirm that in the latest master, using linux-yocto-dev and qemuarm, we no longer get the .config skew when building linux-yocto followed by kernel-devsrc.

build [/home/bruc...poky/build]> diff -u tmp/work/qemuarm-poky-linux-gnueabi/linux-yocto-dev/6.2+gitAUTOINC+bd77e1f904_06afce40af-r0/linux-qemuarm-standard-build/.config tmp/work-shared/qemuarm/kernel-build-artifacts/.config
build [/home/bruc...poky/build]> grep GCC tmp/work/qemuarm-poky-linux-gnueabi/linux-yocto-dev/6.2+gitAUTOINC+bd77e1f904_06afce40af-r0/linux-qemuarm-standard-build/.config
CONFIG_CC_VERSION_TEXT="arm-poky-linux-gnueabi-gcc (GCC) 12.2.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=120200
CONFIG_GCC11_NO_ARRAY_BOUNDS=y
CONFIG_GCC12_NO_ARRAY_BOUNDS=y
CONFIG_HAVE_GCC_PLUGINS=y
CONFIG_GCC_PLUGINS=y
# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
build [/home/bruc...poky/build]> grep GCC tmp/work-shared/qemuarm/kernel-build-artifacts/.config
CONFIG_CC_VERSION_TEXT="arm-poky-linux-gnueabi-gcc (GCC) 12.2.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=120200
CONFIG_GCC11_NO_ARRAY_BOUNDS=y
CONFIG_GCC12_NO_ARRAY_BOUNDS=y
CONFIG_HAVE_GCC_PLUGINS=y
CONFIG_GCC_PLUGINS=y
# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set

But we still have the SDK and kernel module problem as the gcc plugins don't have all required dependencies.

For completeness, another past mention from the list:
https://lists.openembedded.org/g/openembedded-core/topic/89633477

@Bruce, I take it the issue with gcc plugins in the SDK is not fixed yet?

Yes, the SDK part is still outstanding.

I was leaving it as-is for now, to see if anyone else reported the issue (and use that to bump the priority).

I have another SDK bug that I was looking into, and could poke at this more now

Are you experiencing the same issue ? Anything new to add for a reproducing set of steps ?

Yes, the same SDK issue with gcc-plugins building 32-bit ARM kernel:

> cc1: error: cannot load plugin ./scripts/gcc-plugins/arm_ssp_per_task_plugin.so: ./scripts/gcc-plugins/arm_ssp_per_task_plugin.so: undefined symbol: _ZN8opt_pass14set_pass_paramEjb

This happens with gcc11 based toolchain from OE SDK built with Kirkstone. The kernel is 6.1 LTS (seeing it with TI BSP kernel, but should be the same with vanilla):

> make multi_v7_defconfig
> make zImage

That defconfig enables these config options in question:

> CONFIG_HAVE_GCC_PLUGINS=y
> CONFIG_GCC_PLUGINS=y
> CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK=y

Now, older kernel (say 5.10) had a more robust check for gcc-plugins actually working, so it would end up implicitly skipping them when building, effectively disabling CONFIG_GCC_PLUGINS completely (and skipping SSP_PER_TASK specifically).

But that got "simplified" in the commit linked by Sumit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e860048c53ee77ee9870dcce94847a28544b753

And now it just checks for the presence of the plugin-version.h header, which doesn't seem enough... Well, all that was already documented here more than a year ago by Daniel. And later, Jeroen posted a patch to attempt to fix it. His "rdynamic" comments are interesting. But requiring to bundle a matching host gcc with the cross SDK seems unreasonable. It does sound like something is missing from our cross SDK to get the gcc-plugins working properly...

One more data point - I'm not seeing this issue with the latest gcc13 SDK from master. The same kernel builds fine, but the output from "make multi_v7_defconfig" is different - the resulting .config does have generic CONFIG_HAVE_GCC_PLUGINS and CONFIG_GCC_PLUGINS options enabled, but there's no mention of the problematic CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK in there at all. So, it's being skipped and everything builds fine.

Kirkstone SDK has gcc 11.4.0 and master SDK has gcc 13.2.0. I do see that both SDKs have the required plugin-version.h file. Testing this on Ubuntu 22.04, which also has gcc 11.4.0 as a host gcc. I also tried this on Ubuntu 20.04 with an older gcc 9, then reading that this could be due to gcc mismatch between host and cross compiler, updated gcc to 11.4. But that didn't help or changed anything.

Here's a diff between "make multi_v7_defconfig" output of Kirkstone SDK and master SDK:


@@ -2,14 +2,14 @@
 # Automatically generated file; DO NOT EDIT.
 # Linux/arm 6.1.46 Kernel Configuration
 #
-CONFIG_CC_VERSION_TEXT="arm-oe-linux-gnueabi-gcc (GCC) 11.4.0"
+CONFIG_CC_VERSION_TEXT="arm-oe-linux-gnueabi-gcc (GCC) 13.2.0"
 CONFIG_CC_IS_GCC=y
-CONFIG_GCC_VERSION=110400
+CONFIG_GCC_VERSION=130200
 CONFIG_CLANG_VERSION=0
 CONFIG_AS_IS_GNU=y
-CONFIG_AS_VERSION=20244508
+CONFIG_AS_VERSION=24100
 CONFIG_LD_IS_BFD=y
-CONFIG_LD_VERSION=20244508
+CONFIG_LD_VERSION=24100
 CONFIG_LLD_VERSION=0
 CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
 CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y
@@ -675,6 +675,7 @@
 # CONFIG_PARAVIRT is not set
 # CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
 # CONFIG_XEN is not set
+CONFIG_CC_HAVE_STACKPROTECTOR_TLS=y
 CONFIG_STACKPROTECTOR_PER_TASK=y
 # end of Kernel Features
 
@@ -910,7 +911,6 @@
 CONFIG_HAVE_GCC_PLUGINS=y
 CONFIG_GCC_PLUGINS=y
 # CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
-CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK=y
 # end of General architecture-dependent options
 
 CONFIG_RT_MUTEXES=y
@@ -9271,10 +9271,12 @@
 #
 # Memory initialization
 #
-CONFIG_INIT_STACK_NONE=y
-# CONFIG_GCC_PLUGIN_STRUCTLEAK_USER is not set
-# CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF is not set
-# CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL is not set
+CONFIG_CC_HAS_AUTO_VAR_INIT_PATTERN=y
+CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO_BARE=y
+CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO=y
+# CONFIG_INIT_STACK_NONE is not set
+# CONFIG_INIT_STACK_ALL_PATTERN is not set
+CONFIG_INIT_STACK_ALL_ZERO=y
 # CONFIG_INIT_ON_ALLOC_DEFAULT_ON is not set
 # CONFIG_INIT_ON_FREE_DEFAULT_ON is not set
 CONFIG_CC_HAS_ZERO_CALL_USED_REGS=y




Notable 2 changes:

+CONFIG_CC_HAVE_STACKPROTECTOR_TLS=y

-CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK=y

> Notable 2 changes:
> 
> +CONFIG_CC_HAVE_STACKPROTECTOR_TLS=y
> -CONFIG_GCC_PLUGIN_ARM_SSP_PER_TASK=y

That certainly makes sense. These two features are essentially equivalent[1] so as soon as we build with a compiler newer than gcc-12 then the compiler option will be preferred over using a compiler plugin to implement this.

The result is that arm defconfig will no longer enable plugins by default meaning the problems with the SDK become much harder to trigger (*and* we get better stack canary values too ;-) ).


[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f05eb1d24eb564488014d90b31824f71984d7ef5

Bulk move of medium importance bugs from 5.1 to 5.2 on behalf of Bruce.

Moving off old milestone per bug cleanup (to 6.2 M2).