Bug 15493

Summary: sstate reuse not as ideal as it should be
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: coreAssignee: Unassigned <unassigned>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Undecided CC: meta.mr.watcher, meta.watcher, ross.burton, tgamblin
Version: 0.0.0   
Target Milestone: 0.0.0   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2024-05-24 13:59:50 UTC 
I had a world build complete then came back and ran another build the next day having only made bitbake changes:

https://valkyrie.yoctoproject.org/#/builders/17/builds/18/steps/11/logs/stdio

Sstate summary: Wanted 19990 Local 17552 Mirrors 0 Missed 2438 Current 0 (87% match, 0% complete)

There is a lot of sstate matching but there were also rebuilds of base-files, connman, bluez, linux-yocto and more which was unexpected.

Theories:
    * the revision changed, changing base-files and things have a dependency on base-files, maybe useradd related which ripple causing more rebuilds than they should

    * somehow DATE/TIME may also come into play

We should invesitgate the dependencies and fix them so we can see better sstate reuse.
Comment 1 Ross Burton 2024-05-24 14:39:43 UTC 
I suspect I've seen this before but not reliably. My hunch was the DISTRO_VERSION gets into the dependency tree somewhow.
Comment 2 Richard Purdie 2024-05-24 14:49:35 UTC 
https://valkyrie.yoctoproject.org/#/builders/17/builds/19/steps/12/logs/stdio

was a following build without changing anything. sstate reuse is better but some things were still rebuilding (why?)

I then changed the master-next revision and started:

https://valkyrie.yoctoproject.org/#/builders/17/builds/20
Comment 3 Richard Purdie 2024-05-24 14:55:30 UTC 
Changing the master-next revision dropped things back to:

Sstate summary: Wanted 19990 Local 17552 Mirrors 0 Missed 2438 Current 0 (87% match, 0% complete)

when the repeat build was 98%
Comment 4 Richard Purdie 2024-05-24 15:20:01 UTC 
This is probably systemd depending on os-release. I'm testing a tweak for that but there may be further issues.
Comment 5 Ross Burton 2024-05-24 15:39:02 UTC 
I did a build of systemd.  Touched and committed a file in poky, and built systemd:

NOTE: recipe base-files-3.0.14-r0: task do_install: Started
NOTE: recipe os-release-1.0-r0: task do_compile: Started
NOTE: recipe os-release-1.0-r0: task do_install: Started
NOTE: recipe base-files-3.0.14-r0: task do_populate_sysroot: Started
NOTE: recipe base-files-3.0.14-r0: task do_package: Started
NOTE: recipe os-release-1.0-r0: task do_package: Started
NOTE: recipe os-release-1.0-r0: task do_packagedata: Started
NOTE: recipe base-files-3.0.14-r0: task do_packagedata: Started
NOTE: recipe os-release-1.0-r0: task do_create_spdx: Started
NOTE: recipe os-release-1.0-r0: task do_package_write_ipk: Started
NOTE: recipe systemd-1_255.4-r0: task do_package_write_ipk_setscene: Started
NOTE: recipe systemd-1_255.4-r0: task do_package_qa_setscene: Started
NOTE: recipe systemd-1_255.4-r0: task do_create_runtime_spdx: Started

So nobody _really_ notices because systemd only repackages, but this is definitely a path where SHAs changing causes rebuilds.
Comment 6 Richard Purdie 2024-05-30 07:54:27 UTC 
(In reply to Ross Burton from comment #5)
> I did a build of systemd.  Touched and committed a file in poky, and built
> systemd:
> 
> NOTE: recipe base-files-3.0.14-r0: task do_install: Started
> NOTE: recipe os-release-1.0-r0: task do_compile: Started
> NOTE: recipe os-release-1.0-r0: task do_install: Started
> NOTE: recipe base-files-3.0.14-r0: task do_populate_sysroot: Started
> NOTE: recipe base-files-3.0.14-r0: task do_package: Started
> NOTE: recipe os-release-1.0-r0: task do_package: Started
> NOTE: recipe os-release-1.0-r0: task do_packagedata: Started
> NOTE: recipe base-files-3.0.14-r0: task do_packagedata: Started
> NOTE: recipe os-release-1.0-r0: task do_create_spdx: Started
> NOTE: recipe os-release-1.0-r0: task do_package_write_ipk: Started
> NOTE: recipe systemd-1_255.4-r0: task do_package_write_ipk_setscene: Started
> NOTE: recipe systemd-1_255.4-r0: task do_package_qa_setscene: Started
> NOTE: recipe systemd-1_255.4-r0: task do_create_runtime_spdx: Started
> 
> So nobody _really_ notices because systemd only repackages, but this is
> definitely a path where SHAs changing causes rebuilds.

Debian renaming dependencies will mean anything with a runtime dependency on systemd would also repackage so this does cause a lot of rebuilding of packages so it is best fixed!