| Summary: | buildperf: Refusing to aggregate buildstats, recipe version differs (linux-yocto) | ||
|---|---|---|---|
| Product: | [QA/Testing] Build Testing | Reporter: | Mathieu Dubois-Briand <mathieu.dubois-briand> |
| Component: | general | Assignee: | Unassigned <unassigned> |
| Status: | NEW --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | pacopedraza87, randy.macleod, ross.burton |
| Version: | unspecified | ||
| Target Milestone: | 6.1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
buildperf-debian11 perf-debian12-vk master completed at 2026-01-03 00:01:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2018/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-03 00:07:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2019/steps/13/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-03 05:22:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2020/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-03 05:21:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2019/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-03 11:22:40+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2021/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-03 11:22:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2020/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-03 17:22:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2022/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-03 17:22:25+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2021/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-03 23:22:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2023/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-03 23:22:13+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2022/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-04 05:22:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2024/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-04 05:22:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2023/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-04 11:22:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2025/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-04 11:22:09+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2024/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-04 20:45:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2027/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-04 20:44:58+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2026/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-04 23:22:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2028/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-04 23:21:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2027/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-05 05:22:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2029/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-05 05:20:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2028/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-05 11:22:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2030/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-05 11:20:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2029/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-05 17:22:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2031/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-05 17:20:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2030/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-05 23:22:31+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2032/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-05 23:20:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2031/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-06 05:22:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2033/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-06 05:20:09+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2032/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-06 11:22:33+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2034/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-06 11:20:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2033/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-06 17:22:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2035/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-06 17:20:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2034/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-06 23:23:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2036/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-06 23:20:56+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2035/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-07 05:23:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2037/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-07 05:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2036/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-07 11:22:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2038/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-07 11:20:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2037/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-07 17:23:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2039/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-07 17:21:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2038/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-07 23:22:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2040/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-07 23:20:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2039/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-08 05:22:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2041/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-08 05:20:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2040/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2026-01-08 11:22:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/2042/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2026-01-08 11:21:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/2041/steps/12/logs/stdio Buildperf builds have gone green again on https://autobuilder.yoctoproject.org/valkyrie/#/builders/57 & https://autobuilder.yoctoproject.org/valkyrie/#/builders/84, and we understand the cause of the issue, so this doesn't need to block 6.0 M1. In the halcyon days of old, we tracked buildperf stats against poky commits. Changes in meta-yocto resulted in the poky commit hash changing. Now, we track buildperf stats against openembedded-core commits as we no longer use the poky combo layer repository. Changes in meta-yocto do not result in the openembedded-core commit hash changing. So two buildperf runs with the same openembedded-core commit hash may be testing different metadata. On Jan 2nd, a commit was pushed to the master branch of meta-yocto to change the default kernel version in poky.conf (https://git.yoctoproject.org/meta-yocto/commit/?id=8ebdf9c45957a5f98db93384992e701f2f01f44c). As there were now multiple buildperf runs with the same openembedded-core commit hash but different default kernel versions, the aggregation of buildstats started to fail. On Jan 5th, a commit was pushed to the master branch of openembedded-core. Now all builds with the latest openembedded-core commit hash used the same kernel version. But, our buildperf jobs compare results for the latest two commits, so bad data was still being loaded for the previous commit. On Jan 8th, another commit was pushed to the master branch of openembedded-core. Now both the latest and previous openembedded-core commit hashes have good data and buildperf jobs are succeeding again. What we need to do is make the buildperf jobs more robust so that they can successfully handle cases like this in the future. |
INFO: Parsing buildstats from 'refs/notes/buildstats/perf-alma8-vk/master/qemux86' Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/buildperf-alma8/build/layers/openembedded-core/scripts/oe-build-perf-report", line 623, in <module> sys.exit(main()) ~~~~^^ File "/srv/pokybuild/yocto-worker/buildperf-alma8/build/layers/openembedded-core/scripts/oe-build-perf-report", line 610, in main buildstats = get_buildstats(repo, notes_ref, notes_ref2, [rev_l, rev_r], outdir) File "/srv/pokybuild/yocto-worker/buildperf-alma8/build/layers/openembedded-core/scripts/oe-build-perf-report", line 430, in get_buildstats buildstats[rev.commit_number][measurement].aggregate(_bs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^ File "/srv/pokybuild/yocto-worker/buildperf-alma8/build/layers/openembedded-core/scripts/lib/buildstats.py", line 283, in aggregate self[pkg].aggregate(data) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "/srv/pokybuild/yocto-worker/buildperf-alma8/build/layers/openembedded-core/scripts/lib/buildstats.py", line 164, in aggregate raise ValueError("Refusing to aggregate buildstats, recipe version " "differs: {} vs. {}".format(self.nevr, bsrecipe.nevr)) ValueError: Refusing to aggregate buildstats, recipe version differs: linux-yocto-6.16.11+git-r0 vs. linux-yocto-6.18.1+git-r0 Running on perf-alma8-vk Using build dir /srv/pokybuild/yocto-worker/buildperf-alma8/build/build