Bug 1525

I'd like to keep this specific to the kernel. Just to clarify what we're talking about here, the problem is when you build a 32 bit userspace but add 64 bit multilibs. Somehow we need to change the kernel we generate to 64 bit. I'm hoping with linux-yocto, this should be as straight foward as changing the KMACHINE from a 32 bit to a 64 bit one? If so, we just need to figure out how to trigger that change and document how to do it. Any comments Bruce?

Hopefully, this is done in such a way it works such that we can promote other recipes to 64bit as well.

(In reply to comment #2)
> Hopefully, this is done in such a way it works such that we can promote other
> recipes to 64bit as well.

Or least other *simple* linux kernel recipes - right now don't use the linux-yocyo tree just yet and might not ever for certain releases we make.

This is trivial for the linux-yocto kernel(s). But you still need to know the
configuration that you'd like to add. i.e. just changing it to CONFIG_64BIT is not
always enough, at times there are other changes that you may or may not need to 
make.

If you want to bump a 32 bit kernel to 64 bit, you can either embed a 64 bit fragment in the meta branch and just add:

  KERNEL_FEATURES_append=" cfg/64bit.scc"

And you'd be done. If you don't have it in the tree, just add a cfg fragment
onto the SRC_URI that has the same contents as you'd have in the in-tree
variant.

This could be done via python, shell, variable, whatever you want to
trigger the promotion.

(In reply to comment #4)
> This is trivial for the linux-yocto kernel(s). But you still need to know the
> configuration that you'd like to add. i.e. just changing it to CONFIG_64BIT is
> not
> always enough, at times there are other changes that you may or may not need to 
> make.
> 
> If you want to bump a 32 bit kernel to 64 bit, you can either embed a 64 bit
> fragment in the meta branch and just add:
> 
>   KERNEL_FEATURES_append=" cfg/64bit.scc"
> 
> And you'd be done. If you don't have it in the tree, just add a cfg fragment
> onto the SRC_URI that has the same contents as you'd have in the in-tree
> variant.
> 
> This could be done via python, shell, variable, whatever you want to
> trigger the promotion.

And if we just added multilib to overrides we could fixup each recipe based off that value.

Does just adding CONFIG_64BIT work for other arches? There are no toolchain issues?

-M

I've used the CONFIG_64

scratch that last comment, it decided to post itself :)

I've used the CONFIG_64BIT 'trick' on most arches and boards that support
64 bit kernels, but sometimes have base 32 bit BSPs. Same thing for endian
swaps.

Even if it is more involved than just CONFIG_64BIT, the arch can be picked up in the fragments and secondary options added.

(In reply to comment #8)
> scratch that last comment, it decided to post itself :)
> 
> I've used the CONFIG_64BIT 'trick' on most arches and boards that support
> 64 bit kernels, but sometimes have base 32 bit BSPs. Same thing for endian
> swaps.
> 
> Even if it is more involved than just CONFIG_64BIT, the arch can be picked
> up in the fragments and secondary options added.

There are actually toolchain issues on powerpc with doing this...

Is see there is a DISTRO_FEATURE = multiarch, and it's fixing some issues (with some recipe hacks) but now I'm hitting some linker issues... still investigating.

-M

(In reply to comment #9)
> (In reply to comment #8)
> > scratch that last comment, it decided to post itself :)
> > 
> > I've used the CONFIG_64BIT 'trick' on most arches and boards that support
> > 64 bit kernels, but sometimes have base 32 bit BSPs. Same thing for endian
> > swaps.
> > 
> > Even if it is more involved than just CONFIG_64BIT, the arch can be picked
> > up in the fragments and secondary options added.
> 
> There are actually toolchain issues on powerpc with doing this...
> 
> Is see there is a DISTRO_FEATURE = multiarch, and it's fixing some issues
> (with some recipe hacks) but now I'm hitting some linker issues... still
> investigating.
> 
> -M

Looks like I just needed this in my kernel recipe:

+KERNEL_CC_append_e6500 = " -m64"
+KERNEL_LD_append_e6500 = " -melf64ppc"

Which is not that surprising considering the default for the 32-bit toolchain is 32 bit stuff... I'm still sort of curious how the CodeSourcery toolchains seem to get away without specifying this...

-M

(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #8)
> > > scratch that last comment, it decided to post itself :)
> > > 
> > > I've used the CONFIG_64BIT 'trick' on most arches and boards that support
> > > 64 bit kernels, but sometimes have base 32 bit BSPs. Same thing for endian
> > > swaps.
> > > 
> > > Even if it is more involved than just CONFIG_64BIT, the arch can be picked
> > > up in the fragments and secondary options added.
> > 
> > There are actually toolchain issues on powerpc with doing this...
> > 
> > Is see there is a DISTRO_FEATURE = multiarch, and it's fixing some issues
> > (with some recipe hacks) but now I'm hitting some linker issues... still
> > investigating.
> > 
> > -M
> 
> Looks like I just needed this in my kernel recipe:
> 
> +KERNEL_CC_append_e6500 = " -m64"
> +KERNEL_LD_append_e6500 = " -melf64ppc"
> 
> Which is not that surprising considering the default for the 32-bit
> toolchain is 32 bit stuff... I'm still sort of curious how the CodeSourcery
> toolchains seem to get away without specifying this...
> 
> -M

Ooooh spoke too soon, not we have kernel module issues...

ERROR: QA Issue: Architecture did not match (20 to 21) on /work/t4240qds-poky-linux/linux-qoriq-sdk-3.0.34-r6/packages-split/kernel-module-sctp/lib/modules/3.0.34-rt55/kernel/net/sctp/sctp.ko

Should be easy enough to work around.

-M

I've seen both of these in the past, but the solution has always
depended on the kernel .. so yes, getting something generic, and 
kernel version independent is the trick.

As for the -m64, I've added it to KERNEL_CC or we've patched the
kernel Makefiles to be a bit smarter and specify it themselves.
Typically it is only the arch/boot code that gets tied up with this.

The QA warning, I've had much more recent experience. Since oe-core
doesn't expect the 32 -> 64 bit mismatch, you can either mark all the
modules with INSANE_SKIP (which isn't necessarily trivial) or you can
modify the insane class to have a kernel exception in the checking
(there's already one for kernel + x32) .. but that exception is a
bit tricky since non kernel-packaged (external) modules will be missed
and from what I've been told, the oe-core design really expects a 
separate 64 bit kernel to be built from the 32 bit userspace (I think
I have that right). But one would think that the multilib support could
trigger an exception to this test, I just haven't found a clean way
to do it yet.

(In reply to comment #12)
> I've seen both of these in the past, but the solution has always
> depended on the kernel .. so yes, getting something generic, and 
> kernel version independent is the trick.
> 
> As for the -m64, I've added it to KERNEL_CC or we've patched the
> kernel Makefiles to be a bit smarter and specify it themselves.
> Typically it is only the arch/boot code that gets tied up with this.

Hmmmm, OK.
 
> The QA warning, I've had much more recent experience. Since oe-core
> doesn't expect the 32 -> 64 bit mismatch, you can either mark all the
> modules with INSANE_SKIP (which isn't necessarily trivial) or you can
> modify the insane class to have a kernel exception in the checking
> (there's already one for kernel + x32) .. but that exception is a
> bit tricky since non kernel-packaged (external) modules will be missed
> and from what I've been told, the oe-core design really expects a 
> separate 64 bit kernel to be built from the 32 bit userspace (I think
> I have that right). But one would think that the multilib support could
> trigger an exception to this test, I just haven't found a clean way
> to do it yet.

This one we can probably work around with the gcc multiarch functions already present, and the qa test and add a proper exception for kernel-modules.

-M

We have done some things for this in meta-fsl-ppc and they are probably hacky but seem to work OK. We added a BUILD_64BIT_KERNEL variable to promote the kernel to a 64-bit variant on a 32-bit machine. 

I'm just asking to look at and keep these in mind, I'm planning on joining any discussion that takes place to solve this problem permanently.

-M

I had a look at meta-fsl-ppc and will definitely keep that approach in mind, and it's similar to 
what I've done in the past:

KERNEL_EXTRA_CC_OPTIONS ?= "${@bb.utils.contains("TRANSLATED_TARGET_ARCH", "powerpc64", "-m64", "", d)}"
KERNEL_EXTRA_LD_OPTIONS ?= "${@bb.utils.contains("TRANSLATED_TARGET_ARCH", "powerpc64", "-melf64ppc", "", d)}"

(assuming that KERNEL_EXTRA_CC_OPTIONS is available).

The inhibiting of the QA check is also interesting. I also do it via variables, but had to do
extra work to pickup modules.

How are you inhibiting module checks ? Is it the all_qa (I'm not completely familiar with it from
memory).

But definitely, good input. We likely want to make the QA checks smarter, with some flags to allow
certain mismatches, but this is a good starting point.

Moving this to 1.4. 

I'm also thinking that this may get into some kernel specific areas, do we want someone
from the kernel "team" to be the assignee for this in 1.4 ? Or is this already well underway ?

(In reply to comment #15)
> I had a look at meta-fsl-ppc and will definitely keep that approach in mind,
> and it's similar to 
> what I've done in the past:
> 
> KERNEL_EXTRA_CC_OPTIONS ?= "${@bb.utils.contains("TRANSLATED_TARGET_ARCH",
> "powerpc64", "-m64", "", d)}"
> KERNEL_EXTRA_LD_OPTIONS ?= "${@bb.utils.contains("TRANSLATED_TARGET_ARCH",
> "powerpc64", "-melf64ppc", "", d)}"
> 
> (assuming that KERNEL_EXTRA_CC_OPTIONS is available).

Probably is available, I used others probably because I saw that method first

> The inhibiting of the QA check is also interesting. I also do it via
> variables, but had to do
> extra work to pickup modules.
> 
> How are you inhibiting module checks ? Is it the all_qa (I'm not completely
> familiar with it from
> memory).

Presumably by removing one of the two being removed. I stopped doing a lot of investigation after the error went away ;)

> But definitely, good input. We likely want to make the QA checks smarter,
> with some flags to allow
> certain mismatches, but this is a good starting point.

Agreed, x32 has some of the same issues with specific exceptions for it.

(In reply to comment #16)
> Moving this to 1.4. 
> 
> I'm also thinking that this may get into some kernel specific areas, do we
> want someone
> from the kernel "team" to be the assignee for this in 1.4 ? Or is this
> already well underway ?

As far as I know there is no work underway for this. Is there anybody from kernel team that can jump in? I'm trying a workload estimation for 1.4 for this task and I don't have a starting point. It seems to me more like a kernel issue, or am I wrong.

It is more of a kernel issue. I'd estimate that this is  10 perfect days work, due to the
time required to discuss and try the various solutions.

I'm adding Darren as the assignee for now, just so he's  aware of this as well.

I meant "Adding Darren to the cc"

Bruce

I can try to help with this, either some specific task or just testing this out... let me know.

-M

bumping this out.  will kick up a design thread once some other issues for M4 have been dealt with (like a new kernel version).

We have some ideas for this, but it's too late for 1.4, the status quo of insane skip will
stay for now. The changes this late in the -dev cycle would carry enough risk to make
bumping this a better idea than pushing it into the release.

moving to 1.6, will do this early in the new development cycle.

Hi, 

This actually looks like something that will need to be documentezasd in the YP user manuals once a solution is figured out.  I am noting that here mostly for myself so I don't lose sight of it.

Scott

This is not a common use-case so the user should take responsibilty for that proper kernel configuration.