Bug 15471

Hello,

I've encountered this very niche bug involving .rpm dependency issues while using the SSTATE cache with out-of-tree kernel module recipes and the LOCALVERSION_AUTO kernel config. Several steps are required to reproduce it which made it hard to define. I could reproduce this issue with kirkstone as well as master poky branches.

Error log:

Bitbake will print out this kind of error after very specific steps described bellow when the LOCALVERSION_AUTO kernel configuration is enabled:

ERROR: image-prod-1.0-r0 do_rootfs: Could not invoke dnf. Command '*****' returned 1:
Error: 
 Problem: package kernel-module-nxp89xx-1.0-r0.imx8m_mini requires kernel-module-moal-5.15.32-mcm-imx8m-mini-1.1-00133-g8f782478016f, but none of the providers can be installed
  - conflicting requests
  - nothing provides kernel-5.15.32-mcm-imx8m-mini-1.1-00133-g8f782478016f needed by kernel-module-moal-5.15.32-mcm-imx8m-mini-1.1-00133-g8f782478016f-1.0-r0.premiere_imx8m_mini
  - nothing provides kernel-module-cfg80211-5.15.32-mcm-imx8m-mini-1.1-00133-g8f782478016f needed by kernel-module-moal-5.15.32-mcm-imx8m-mini-1.1-00133-g8f782478016f-1.0-r0.premiere_imx8m_mini


Steps to reproduce:

The first mecanism that causes this error is a lack of reproducibility in the kernel package version when CONFIG_LOCALVERSION_AUTO is enabled. If at least one patch is present in the recipe, every time the kernel is re-patched, the patch is commited with a new commit number (not the same date!). kernel.bbclass will then extract the KERNEL_VERSION from the kernel script setlocalversion which will point to that temporary commit. The generated .rpm package will explicitly contain this commit SHA in it's version number.

If the generated image contains external modules recipes (like lttng modules), they will be packaged against this kernel package with the specific commit SHA. In a direct invokation, this works well since the dependency between the .rpm module packages and the kernel use the same commit SHA.

However, if the user forces to rebuil the kernel, for instance `bitbake virtual/kernel -c cleansstate`, then recompiles a new image, then a new commit SHA is generated. The external modules however do not need to be re-compiled according to the SSTATE algorithm. They still point to the old temporary SHA though!

Hence, during the rootfs task, dnf will not be able to resolve the version dependency between external modules and the kernel.

So to sum it up:
 - Have at least one kernel patch file in the kernel's SRC_URI
 - Enable CONFIG_LOCALVERSION_AUTO in the kernel configuration
 - Install at least one external module in the image (Ex: lttng-modules)
 - Build the image a first time (Ex: bitbake core-image-minimal)
 - Force recompiling only the kernel (Ex: bitbake virtual/kernel -c cleansstate)
 - Build the image a second time (with master, it is required to also modify the image recipe to prevent hash equivalence to skip running do_rootfs again)

Workarounds:

Any of these will work:
 - Invalidating the kernel SSTATE (doesn't even need to modify the sources): bitbake virtual/kernel -C patch
 - disabling the LOCALVERSION_AUTO kernel configuration

Proper fixing:

I haven't yet analyzed how this could be fixed and my knowledge of the related SSTATE mechanisms is too limited. Several ideas that could resolve the issue:
 - Make the kernel version reproducible with CONFIG_LOCALVERSION_AUTO
 - Always invalidate the kernel modules SSTATE when the kernel is rebuilt

I don't think this bug is critical since it's very specific and workarounds are easy to set up and non intrusive. Still I think it's important for me to document it so that users who may stumble upon this error can properly understand it.

Users are not really expected to force recompiling the kernel. However this has happened several times to me especially with exotic layers, and CIs. In theory I would expect the shared state algorithm to not produce this situation where calling bitbake will fail because of sstate version misalignment.