Bug 15898

Summary:	New sstate.yp.org CDN revealed partial sstate reproducibility tests false-positive error
Product:	[QA/Testing] Functional (self) Testing	Reporter:	Yoann Congal <yoann.congal>
Component:	oe-selftest	Assignee:	Yoann Congal <yoann.congal>
Status:	RESOLVED NOTABUG	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	richard.purdie, tim.orling
Version:	unspecified
Target Milestone:	5.3 M2
Hardware:	x86
OS:	Multiple
See Also:	https://bugzilla.yoctoproject.org/show_bug.cgi?id=15701
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Yoann Congal 2025-06-09 22:31:31 UTC

When trying to do reproducibility test with partial sstate (e.g. with OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS or OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS), build succeed but the test ultimately fails because logs contains non-fatal "No sstate archive obtainable, will run full task instead" ERROR:
2025-06-09 10:08:27,803 - oe-selftest - INFO -       WARNING: quilt-native-0.69-r0 do_recipe_qa_setscene: Failed to fetch URL file://universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo;downloadfilename=universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo, attempting MIRRORS if available
2025-06-09 10:08:27,803 - oe-selftest - INFO -       ERROR: quilt-native-0.69-r0 do_recipe_qa_setscene: Fetcher failure: Unable to find file file://universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo;downloadfilename=universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo anywhere to download to $TOP/builds/build-qemux86-64-st/reproducibleB-extended/tmp/sstate/universal/66/08/sstate:quilt-native:x86_64-linux:0.69:r0:x86_64:14:660833943994a8cab2a929ec5bb0bdb4e1c70ab8df060e87aded0185c7b03cf4_recipe_qa.tar.zst.siginfo. The paths that were searched were:
2025-06-09 10:08:27,803 - oe-selftest - INFO -           $TOP/builds/build-qemux86-64-st/reproducibleB-extended/tmp/sstate
2025-06-09 10:08:27,803 - oe-selftest - INFO -       WARNING: quilt-native-0.69-r0 do_recipe_qa_setscene: No sstate archive obtainable, will run full task instead.
2025-06-09 10:08:27,803 - oe-selftest - INFO -       WARNING: Logfile for failed setscene task is $TOP/builds/build-qemux86-64-st/reproducibleB-extended/tmp/work/x86_64-linux/quilt-native/0.69/temp/log.do_recipe_qa_setscene.2547074
2025-06-09 10:08:27,803 - oe-selftest - INFO -       WARNING: Setscene task ($TOP/builds/build-qemux86-64-st/../../layers/poky/meta/recipes-devtools/quilt/quilt-native_0.69.bb:do_recipe_qa_setscene) failed with exit code '1' - real task will be run instead
...
2025-06-09 10:08:27,804 - oe-selftest - INFO -       NOTE: Tasks Summary: Attempted 888 tasks of which 0 didn't need to be rerun and all succeeded.
2025-06-09 10:08:27,804 - oe-selftest - INFO -       NOTE: Generating JSON CVE summary
2025-06-09 10:08:27,804 - oe-selftest - INFO -       Complete CVE JSON report summary created at: /home/yocon/Documents/projets/yocto/contribution-yocto/builds/build-qemux86-64-st/reproducibleB-extended/tmp/log/cve/cve-summary.json
2025-06-09 10:08:27,804 - oe-selftest - INFO -       
2025-06-09 10:08:27,804 - oe-selftest - INFO -       Summary: There were 268 WARNING messages.
2025-06-09 10:08:27,804 - oe-selftest - INFO -       Summary: There were 178 ERROR messages, returning a non-zero exit code.
2025-06-09 10:08:27,805 - oe-selftest - ERROR - reproducibleB-extended build failed. Trying to compute built packages differences but the test will fail.

Note: reproducibility test case does use: 
  SSTATE_MIRRORS = "file://.*/.*-native.*  http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH file://.*/.*-cross.*  http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
# sstate from CDN for -native/-cross packages

That line did not change with the recent jsdeliver->sstate.yp.org CDN change.

My hypothesis:
* the SSTATE_MIRRORS was wrong during the time jsdeliver was the CDN and the sstate code handled it silently by running the real task.
* Now that the URL is more correct (at least the domain name), the CDN is reachable and answer negatively to requests and that triggers the ERROR line.

Ideas:
* Using SSTATE_MIRRORS = "" fixes it (but that mean not using sstate for -native in the default world builds, so not a real solution)
* Maybe use the yp.org hasequiv server as well? (the logs contains a warning about using an sstate server without its hashequiv counter-part)
 * My idea is that it would avoid asking the sstate server for a non-existing hash but would that work in general (non Yocto MACHINEs, not poky DISTRO, ...)?
* https://bugzilla.yoctoproject.org/show_bug.cgi?id=15701#c3 is a fix for OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS (build A with sstate, build B with sstate, remove sstate for LEAF_TARGET, rebuild B)

Comment 1 Yoann Congal 2025-06-09 22:52:49 UTC

Quick reproducer:
  clone poky (tested at f98ab9c342a5b893611a877133bc05ec143519a0)
  . oe-init-build-env build
  echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf
  echo 'OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS = "bash"' >> conf/local.conf
  oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds

Comment 2 Yoann Congal 2025-06-11 14:20:21 UTC

I'm not the only one seeing this:
https://libera.catirclogs.org/yocto/2025-06-11#38273081;

Comment 3 Richard Purdie 2025-06-12 10:09:24 UTC

The old urls did work even when jsdeliver was being used, they just used our servers directly rather than the CDN (which is fine on the autobuilder as they're local).

I'm therefore not sure what changed to now cause errors.

Comment 4 Tim Orling 2025-06-12 14:38:15 UTC

Was the AI bot blocker causing a fetching failure?

Comment 5 Yoann Congal 2025-06-13 07:42:44 UTC

Same test, few days later: No error.

I'll close this for now but I'll note that the reproducibility test does consider any error fatal (even the usually non fatal ones like "can't fetch sstate, will run the real task)")