15898 2025-06-09 22:31:31 +0000 New sstate.yp.org CDN revealed partial sstate reproducibility tests false-positive error 2025-06-13 07:42:44 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified x86 Multiple RESOLVED NOTABUG https://bugzilla.yoctoproject.org/show_bug.cgi?id=15701 Medium+ normal 5.3 M2 1 yoann.congal yoann.congal richard.purdie tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 102220 0 yoann.congal 2025-06-09 22:31:31 +0000 When trying to do reproducibility test with partial sstate (e.g. with OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS or OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS), build succeed but the test ultimately fails because logs contains non-fatal "No sstate archive obtainable, will run full task instead" ERROR: 2025-06-09 10:08:27,803 - oe-selftest - INFO - WARNING: quilt-native-0.69-r0 do_recipe_qa_setscene: Failed to fetch URL file://universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo;downloadfilename=universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo, attempting MIRRORS if available 2025-06-09 10:08:27,803 - oe-selftest - INFO - ERROR: quilt-native-0.69-r0 do_recipe_qa_setscene: Fetcher failure: Unable to find file file://universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo;downloadfilename=universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo anywhere to download to $TOP/builds/build-qemux86-64-st/reproducibleB-extended/tmp/sstate/universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo. The paths that were searched were: 2025-06-09 10:08:27,803 - oe-selftest - INFO - $TOP/builds/build-qemux86-64-st/reproducibleB-extended/tmp/sstate 2025-06-09 10:08:27,803 - oe-selftest - INFO - WARNING: quilt-native-0.69-r0 do_recipe_qa_setscene: No sstate archive obtainable, will run full task instead. 2025-06-09 10:08:27,803 - oe-selftest - INFO - WARNING: Logfile for failed setscene task is $TOP/builds/build-qemux86-64-st/reproducibleB-extended/tmp/work/x86_64-linux/quilt-native/0.69/temp/log.do_recipe_qa_setscene.2547074 2025-06-09 10:08:27,803 - oe-selftest - INFO - WARNING: Setscene task ($TOP/builds/build-qemux86-64-st/../../layers/poky/meta/recipes-devtools/quilt/quilt-native_0.69.bb:do_recipe_qa_setscene) failed with exit code '1' - real task will be run instead ... 2025-06-09 10:08:27,804 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 888 tasks of which 0 didn't need to be rerun and all succeeded. 2025-06-09 10:08:27,804 - oe-selftest - INFO - NOTE: Generating JSON CVE summary 2025-06-09 10:08:27,804 - oe-selftest - INFO - Complete CVE JSON report summary created at: /home/yocon/Documents/projets/yocto/contribution-yocto/builds/build-qemux86-64-st/reproducibleB-extended/tmp/log/cve/cve-summary.json 2025-06-09 10:08:27,804 - oe-selftest - INFO - 2025-06-09 10:08:27,804 - oe-selftest - INFO - Summary: There were 268 WARNING messages. 2025-06-09 10:08:27,804 - oe-selftest - INFO - Summary: There were 178 ERROR messages, returning a non-zero exit code. 2025-06-09 10:08:27,805 - oe-selftest - ERROR - reproducibleB-extended build failed. Trying to compute built packages differences but the test will fail. Note: reproducibility test case does use: SSTATE_MIRRORS = "file://.*/.*-native.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH file://.*/.*-cross.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH" # sstate from CDN for -native/-cross packages That line did not change with the recent jsdeliver->sstate.yp.org CDN change. My hypothesis: * the SSTATE_MIRRORS was wrong during the time jsdeliver was the CDN and the sstate code handled it silently by running the real task. * Now that the URL is more correct (at least the domain name), the CDN is reachable and answer negatively to requests and that triggers the ERROR line. Ideas: * Using SSTATE_MIRRORS = "" fixes it (but that mean not using sstate for -native in the default world builds, so not a real solution) * Maybe use the yp.org hasequiv server as well? (the logs contains a warning about using an sstate server without its hashequiv counter-part) * My idea is that it would avoid asking the sstate server for a non-existing hash but would that work in general (non Yocto MACHINEs, not poky DISTRO, ...)? * https://bugzilla.yoctoproject.org/show_bug.cgi?id=15701#c3 is a fix for OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS (build A with sstate, build B with sstate, remove sstate for LEAF_TARGET, rebuild B) 102221 1 yoann.congal 2025-06-09 22:52:49 +0000 Quick reproducer: clone poky (tested at f98ab9c342a5b893611a877133bc05ec143519a0) . oe-init-build-env build echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf echo 'OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS = "bash"' >> conf/local.conf oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds 102232 2 yoann.congal 2025-06-11 14:20:21 +0000 I'm not the only one seeing this: https://libera.catirclogs.org/yocto/2025-06-11#38273081; 102235 3 richard.purdie 2025-06-12 10:09:24 +0000 The old urls did work even when jsdeliver was being used, they just used our servers directly rather than the CDN (which is fine on the autobuilder as they're local). I'm therefore not sure what changed to now cause errors. 102247 4 tim.orling 2025-06-12 14:38:15 +0000 Was the AI bot blocker causing a fetching failure? 102261 5 yoann.congal 2025-06-13 07:42:44 +0000 Same test, few days later: No error. I'll close this for now but I'll note that the reproducibility test does consider any error fatal (even the usually non fatal ones like "can't fetch sstate, will run the real task)")