| Summary: | buildperf builds failing with bitbake-setup | ||
|---|---|---|---|
| Product: | [Infrastructure] AutoBuilder | Reporter: | Ross Burton <ross.burton> |
| Component: | autobuilder | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | High | CC: | infras.ab.watcher, Infras.watcher, mathieu.dubois-briand, pidge, randy.macleod |
| Version: | unspecified | ||
| Target Milestone: | 5.3 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Ross Burton
2025-10-16 14:27:53 UTC
Since we will no longer be maintaining the poky combo-layer, we need to figure out how to proceed. 1. How can we include the bitbake, and meta-yocto commit ids? Richard generated a new sha1 for buildbot that combines oe-core, bitbake, meta-yocto. We could just concatenate the three hashes. 2. Should we preserve the historic data or re-generate it? 3. Also the performance scripting code is in need of refreshing. https://valkyrie.yocto.io/pub/non-release/20251016-73/testresults/buildperf-debian11/perf-debian12-vk_master_20251016095436_63d586cd64.html shows the problem: theres just a day of build perf data. Presumably this is because it doesn't know that the old poky data should be glued in front of the new oe-core data. The raw data is in git, eg https://git.yoctoproject.org/yocto-buildstats/log/?h=perf-debian12-vk/master/qemux86 buildperf-alma8 perf-alma8-vk master completed at 2025-10-21 11:22:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1708/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-21 11:21:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1707/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-21 17:22:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1709/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-21 17:20:29+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1708/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-21 23:22:47+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1710/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-21 23:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1709/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 05:22:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1711/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 05:21:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1710/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 11:21:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1712/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 11:20:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1711/steps/12/logs/stdio So I had an extensive look at the last failures. Basically, the issue is the same oe-core commit has been built with different versions of bitbake. I believe this happens when on a given day, some commits are merged in bitbake repo but none is merged in oe-core. So the builds from the previous and next nights will both use the same oe-core sha1 as a reference, but the build meta-data will differ. That's why we do get: AssertionError: Data mismatch 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c != 35a421cdcafb3595b9de5489ffdc567825400d26 We are checking that all measurements for a given oe-core version have the same meta-data. Here they do differ on the bitbake revision: 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c on one side, 35a421cdcafb3595b9de5489ffdc567825400d26 on the other. Now what is the correct solution here? - completely ignore differences on bitbake and meta-yocto revisions? - for each oe-core commit, only consider buildperf measurements for the most recent bitbake and meta-yocto commits? - parse all oe-core/bitbake/meta-yocto combinations separately? I'm not really sure how we would handle that, reading the code I believe it's far from a straightforward change. I've been playing with the second solution here, and it looks like it works correctly locally. I'm not sure this is the best solution, but I send some patch to discuss this. buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 17:22:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1713/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 17:20:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1712/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 23:22:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1714/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 23:20:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1713/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 05:22:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1715/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 05:20:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1714/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 11:22:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1716/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 11:21:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1715/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 17:22:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1717/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 17:22:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1716/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 23:21:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1718/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 23:21:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1717/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 05:22:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1719/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 05:21:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1718/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 11:22:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1720/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 11:21:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1719/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 17:22:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1721/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 17:21:17+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1720/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 23:22:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1722/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 23:22:19+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1721/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 05:23:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1723/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 05:21:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1722/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 11:47:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1724/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 11:22:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1723/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 17:22:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1725/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 17:21:40+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1724/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 23:22:25+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1726/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 23:22:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1725/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 05:22:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1727/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 05:22:33+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1726/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 11:22:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1728/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 11:22:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1727/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 17:22:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1729/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 17:21:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1728/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 23:22:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1730/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 23:21:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1729/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 05:22:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1731/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 05:20:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1730/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 11:22:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1732/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 11:21:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1731/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 17:22:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1733/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 17:21:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1732/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 23:22:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1734/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 23:21:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1733/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 05:22:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1735/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 05:21:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1734/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 11:22:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1736/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 11:20:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1735/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 17:27:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1737/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 17:20:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1736/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 23:22:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1738/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 23:20:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1737/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 05:23:05+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1739/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 05:20:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1738/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 11:23:25+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1740/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 11:21:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1739/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 17:22:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1741/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 17:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1740/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 23:22:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1742/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 23:21:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1741/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-30 05:22:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1743/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-30 05:21:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1742/steps/12/logs/stdio |