16014 2025-10-16 14:27:53 +0000 buildperf builds failing with bitbake-setup 2025-11-20 15:49:44 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple RESOLVED FIXED High normal 5.3 M4 1 ross.burton richard.purdie infras.ab.watcher Infras.watcher mathieu.dubois-briand pidge randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 102953 0 ross.burton 2025-10-16 14:27:53 +0000 102968 1 randy.macleod 2025-10-16 15:14:52 +0000 Since we will no longer be maintaining the poky combo-layer, we need to figure out how to proceed. 1. How can we include the bitbake, and meta-yocto commit ids? Richard generated a new sha1 for buildbot that combines oe-core, bitbake, meta-yocto. We could just concatenate the three hashes. 2. Should we preserve the historic data or re-generate it? 3. Also the performance scripting code is in need of refreshing. 102970 2 ross.burton 2025-10-16 15:16:47 +0000 https://valkyrie.yocto.io/pub/non-release/20251016-73/testresults/buildperf-debian11/perf-debian12-vk_master_20251016095436_63d586cd64.html shows the problem: theres just a day of build perf data. Presumably this is because it doesn't know that the old poky data should be glued in front of the new oe-core data. The raw data is in git, eg https://git.yoctoproject.org/yocto-buildstats/log/?h=perf-debian12-vk/master/qemux86 103021 3 mathieu.dubois-briand 2025-10-22 13:03:08 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-21 11:22:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1708/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-21 11:21:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1707/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-21 17:22:38+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1709/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-21 17:20:29+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1708/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-21 23:22:47+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1710/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-21 23:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1709/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 05:22:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1711/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 05:21:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1710/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 11:21:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1712/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 11:20:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1711/steps/12/logs/stdio 103022 4 mathieu.dubois-briand 2025-10-22 13:15:56 +0000 So I had an extensive look at the last failures. Basically, the issue is the same oe-core commit has been built with different versions of bitbake. I believe this happens when on a given day, some commits are merged in bitbake repo but none is merged in oe-core. So the builds from the previous and next nights will both use the same oe-core sha1 as a reference, but the build meta-data will differ. That's why we do get: AssertionError: Data mismatch 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c != 35a421cdcafb3595b9de5489ffdc567825400d26 We are checking that all measurements for a given oe-core version have the same meta-data. Here they do differ on the bitbake revision: 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c on one side, 35a421cdcafb3595b9de5489ffdc567825400d26 on the other. Now what is the correct solution here? - completely ignore differences on bitbake and meta-yocto revisions? - for each oe-core commit, only consider buildperf measurements for the most recent bitbake and meta-yocto commits? - parse all oe-core/bitbake/meta-yocto combinations separately? I'm not really sure how we would handle that, reading the code I believe it's far from a straightforward change. I've been playing with the second solution here, and it looks like it works correctly locally. I'm not sure this is the best solution, but I send some patch to discuss this. 103037 5 ross.burton 2025-10-23 12:28:16 +0000 https://lists.openembedded.org/g/openembedded-core/message/225194 103038 6 mathieu.dubois-briand 2025-10-23 12:33:20 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 17:22:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1713/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 17:20:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1712/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-22 23:22:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1714/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-22 23:20:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1713/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 05:22:51+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1715/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 05:20:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1714/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 11:22:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1716/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 11:21:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1715/steps/12/logs/stdio 103059 7 mathieu.dubois-briand 2025-10-24 07:11:15 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 17:22:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1717/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 17:22:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1716/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-23 23:21:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1718/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-23 23:21:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1717/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 05:22:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1719/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 05:21:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1718/steps/12/logs/stdio 103086 8 mathieu.dubois-briand 2025-10-27 09:51:08 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 11:22:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1720/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 11:21:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1719/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 17:22:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1721/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 17:21:17+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1720/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-24 23:22:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1722/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-24 23:22:19+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1721/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 05:23:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1723/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 05:21:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1722/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 11:47:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1724/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 11:22:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1723/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 17:22:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1725/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 17:21:40+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1724/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-25 23:22:25+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1726/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-25 23:22:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1725/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 05:22:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1727/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 05:22:33+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1726/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 11:22:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1728/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 11:22:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1727/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 17:22:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1729/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 17:21:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1728/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-26 23:22:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1730/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-26 23:21:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1729/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 05:22:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1731/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 05:20:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1730/steps/12/logs/stdio 103118 9 mathieu.dubois-briand 2025-10-28 09:02:44 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 11:22:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1732/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 11:21:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1731/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 17:22:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1733/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 17:21:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1732/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-27 23:22:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1734/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-27 23:21:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1733/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 05:22:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1735/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 05:21:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1734/steps/12/logs/stdio 103132 10 mathieu.dubois-briand 2025-10-29 11:56:49 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 11:22:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1736/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 11:20:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1735/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 17:27:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1737/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 17:20:54+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1736/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-28 23:22:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1738/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-28 23:20:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1737/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 05:23:05+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1739/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 05:20:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1738/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 11:23:25+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1740/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 11:21:15+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1739/steps/12/logs/stdio 103232 11 mathieu.dubois-briand 2025-11-10 09:00:29 +0000 buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 17:22:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1741/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 17:20:48+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1740/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-29 23:22:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1742/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-29 23:21:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1741/steps/12/logs/stdio buildperf-alma8 perf-alma8-vk master completed at 2025-10-30 05:22:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/57/builds/1743/steps/13/logs/stdio buildperf-debian11 perf-debian12-vk master completed at 2025-10-30 05:21:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/84/builds/1742/steps/12/logs/stdio 103351 12 richard.purdie 2025-11-20 15:49:44 +0000 https://git.openembedded.org/openembedded-core/commit/?id=e7dc42e30c76bf0fbb4d3cc019bbec675bac55fa