Bug 14154 - yocto-check-layer fails incorrectly with kernel hash changes
Summary: yocto-check-layer fails incorrectly with kernel hash changes
Status: RESOLVED NOTABUG
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: 3.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.2 M3
Assignee: Unassigned
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-12-17 16:31 UTC by Ross Burton
Modified: 2025-04-17 15:10 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ross Burton 2020-12-17 16:31:48 UTC
When running yocto-check-layer on meta-arm-bsp:

$ yocto-check-layer-wrapper ~/Yocto/meta-arm/meta-arm-bsp --dependency ~/Yocto/meta-arm/meta-arm-toolchain/ ~/Yocto/meta-arm/meta-arm

  File "/home/ross/Yocto/poky/scripts/lib/checklayer/cases/common.py", line 55, in test_signatures
    self.fail('Adding layer %s changed signatures.\n%s' % (self.tc.layer['name'], msg))
AssertionError: Adding layer meta-arm-bsp changed signatures.
516 signatures changed, initial differences (first hash before, second after):
   linux-yocto:do_kernel_metadata: 11533494fc57b12b96af4d56f6adbd3ec0c561db12dd201a18e7fa82bd7e18ad -> 5eac2fbe970822cc59cc8a35986c76b768283a2a5ebd1b38417013653f219a25
      bitbake-diffsigs --task linux-yocto do_kernel_metadata --signature 11533494fc57b12b96af4d56f6adbd3ec0c561db12dd201a18e7fa82bd7e18ad 5eac2fbe970822cc59cc8a35986c76b768283a2a5ebd1b38417013653f219a25
      NOTE: Starting bitbake server...
      Dependency on checksum of file juno-standard.scc was added
      Dependency on checksum of file linux-arm64-ack_5.4.bbappend was added
      Dependency on checksum of file 0001-drm-Add-component-aware-simple-encoder.patch was added
      Dependency on checksum of file 0001-ARM-vexpress-enable-GICv3.patch was added
[ pages more of this ]

This is because meta-arm-bsp has a kernel bbappend that does:

FILESEXTRAPATHS_prepend := "${THISDIR}:${THISDIR}/files:"

My understanding is that this is safe to do, but kernel-yocto.bbclass does this:

def get_dirs_with_fragments(d):
    ...
    extrapathsvalue = (d.getVar("FILESEXTRAPATHS") or "")
    ...

do_kernel_metadata[file-checksums] = " ${@get_dirs_with_fragments(d)}"

Which means that the kernel is depending on files that it isn't going to be using.
Comment 1 Richard Purdie 2020-12-17 17:52:31 UTC
I think we need to determine where/what is adding the dependency on these files. You're right that the kernel should be adding dependencies on the things it needs. Other dependencies may be coming from SRC_URI so what is in SRC_URI for this? I think this will need more investigation to find out where the dependencies on the files is being added, that will then determine how to fix it.
Comment 2 Bruce Ashfield 2020-12-17 19:06:51 UTC
Are any of the mentioned .scc files added to the SRC_URI ? We could make a special case if nothing is added from a directory, we don't hash it.

What we have is the result of a simplified original patch (which i unfortunately cannot find, and has probably been lost).

The hashing feature was created such that indirectly referenced patch/config files (i.e. the .scc is on the SRC_URI, and the .scc file includes it) would trigger a rebuild if they are modified.

We can't insist that all .patch/.cfg files are on the SRC_URI, since it actually is quite common that features are grouped within a .scc file, and has been like that for years.

We also can't easily figure out exactly what .patch and .cfg files are used indirectly, since that information is only known once the kernel meta data has been gathered and the features processed. Much to late for the hash. Unless there's some new programatic way to change the has after the meta-data has run.

Which brings us back to my original question about if there are .scc or .cfg files on the SRC_URI at all ? If not, we can exclude that directory and stop it from adding to the hash.
Comment 3 Armin Kuster 2020-12-24 18:36:42 UTC
triaging
Comment 4 Randy MacLeod 2023-07-26 21:22:03 UTC
Bulk move of 64 bugs to 4.3 M3 after a quick review.
If a bug is actually fixed, please add a commit link and resolve it.
-- Randy for YP bug team.
Comment 5 Trevor Gamblin 2023-09-14 16:37:28 UTC
Moved to M4.
Comment 6 Randy MacLeod 2023-10-30 15:29:10 UTC
Bulk move to 5.0 M1. -- Randy
Comment 7 Randy MacLeod 2023-12-21 23:28:30 UTC
Bulk move of 52 5.0-M1 bugs to M2.
Comment 8 Randy MacLeod 2024-01-25 14:28:50 UTC
Bulk move of bugs owned by "unassigned" from 5.0-M2 to 5.0-M3.
Comment 9 Randy MacLeod 2025-01-30 16:06:37 UTC
Bulk move to 5.2-M3.
Comment 10 Randy MacLeod 2025-01-30 16:17:05 UTC
Bulk move to 5.2-M3 and this time I'll actually specify M3!
Comment 11 Randy MacLeod 2025-03-25 16:45:19 UTC
Using the master branches of poky ( 2d66669b8c ) with meta-arm ( f78c6c0e ):

❯ yocto-check-layer-wrapper ../../meta-arm.git/meta-arm-bsp --dependency ../..//meta-arm.git/meta-arm-toolchain/ ../../meta-arm.git/meta-arm

...

INFO: Summary of results:
INFO: 
INFO: meta-arm-bsp ... PASS
INFO: meta-arm ... PASS
INFO: meta-arm-toolchain ... PASS

Ross, 
Do I need to specify a certain MACHINE in local.conf?
Comment 12 Ross Burton 2025-03-27 21:38:03 UTC
We've worked around it by guarding the FILEEXTRAPATH prepend with overrides.  Making the append global breaks as before:

INFO: test_signatures (common.CommonCheckLayer.test_signatures)
INFO:  ... FAIL

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (21:36:25.276379)
Loading cache...done.
Loaded 1977 entries from dependency cache.
INFO: Traceback (most recent call last):
  File "/home/rosbur01/Yocto/poky-apt/scripts/lib/checklayer/cases/common.py", line 131, in test_signatures
    self.fail('Adding layer %s changed signatures.\n%s' % (self.tc.layer['name'], msg))
AssertionError: Adding layer meta-arm-bsp changed signatures.
127 signatures changed, initial differences (first hash before, second after):
   linux-yocto:do_kernel_metadata: f5a2303eba62896506340be757519d6675bd44772f186514b364075a7a9cb0c7 -> cf7b33677f98ec4b8c6fc7df88a052ab7b88e652d24518267457118629d520e5
      bitbake-diffsigs --task linux-yocto do_kernel_metadata --signature f5a2303eba62896506340be757519d6675bd44772f186514b364075a7a9cb0c7 cf7b33677f98ec4b8c6fc7df88a052ab7b88e652d24518267457118629d520e5
      NOTE: Reconnecting to bitbake server...
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/README.md was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/corstone1000-standard.scc was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/corstone1000/base.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp-standard.scc was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp.scc was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-board.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-cfi.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-drm.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-net.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-rtc.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-serial.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/fvp/fvp-watchdog.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno-standard.scc was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno.scc was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-board.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-devfreq.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-dma.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-drm.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-fb.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-i2c.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-mali-midgard.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-mmc.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-net.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-pci.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-rtc.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-sata.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-serial.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-sound.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-thermal.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/juno/juno-usb.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/virtio-9p.cfg was added
      Dependency on checksum of file ./arm-platforms-kmeta/bsp/arm-platforms/virtio-9p.scc was added
      Dependency on checksum of file ./corstone1000/0001-remoteproc-Add-Arm-remoteproc-driver.patch was added
      Dependency on checksum of file ./corstone1000/0002-arm64-dts-Add-corstone1000-external-system-device-no.patch was added
      Dependency on checksum of file ./corstone1000/0003-dt-bindings-remoteproc-Add-Arm-remoteproc.patch was added
      Dependency on checksum of file ./corstone1000/corstone1000_kernel_debug.cfg was added
      Dependency on checksum of file ./corstone1000/defconfig was added
      Dependency on checksum of file ./corstone1000/extsys.cfg was added
      Dependency on checksum of file ./files/corstone1000/0001-remoteproc-Add-Arm-remoteproc-driver.patch was added
      Dependency on checksum of file ./files/corstone1000/0002-arm64-dts-Add-corstone1000-external-system-device-no.patch was added
      Dependency on checksum of file ./files/corstone1000/0003-dt-bindings-remoteproc-Add-Arm-remoteproc.patch was added
      Dependency on checksum of file ./files/corstone1000/corstone1000_kernel_debug.cfg was added
      Dependency on checksum of file ./files/corstone1000/defconfig was added
      Dependency on checksum of file ./files/corstone1000/extsys.cfg was added
      Dependency on checksum of file ./files/fvp-base/0001-arm64-dts-fvp-Enable-virtio-rng-support.patch was added
      Dependency on checksum of file ./fvp-base/0001-arm64-dts-fvp-Enable-virtio-rng-support.patch was added
      Dependency on checksum of file ./linux-arm-platforms.inc was added
      Dependency on checksum of file ./linux-yocto-dev.bbappend was added
      Dependency on checksum of file ./linux-yocto-rt_%.bbappend was added
      Dependency on checksum of file ./linux-yocto_%.bbappend was added

Now, arguably the failure is actually right - adding new directories to the search path can cause other BSPs to find files that it wouldn't have otherwise.  I'm open to be convinced either way, to be honest.
Comment 13 Ross Burton 2025-04-17 15:10:08 UTC
On further discussion we think that this is the right thing to do: bbappends shouldn't be adding directories to the file search path globally because that will potentially disrupt file lookup for unrelated BSPs.