| Summary: | intermittent setscene tasks failures | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Alexandre Belloni <alexandre.belloni> |
| Component: | oe-core other | Assignee: | Anuj Mittal <anuj.mittal> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alexandre.belloni, anuj.mittal, matt, randy.macleod, richard.purdie, thomas.perrot |
| Version: | 3.4 | ||
| Target Milestone: | 3.4.3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT AB-INT-NET | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Alexandre Belloni
2022-01-08 00:28:23 UTC
https://autobuilder.yoctoproject.org/typhoon/#builders/97/builds/3794/steps/15/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/61/builds/4448/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/50/builds/4459/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/65/builds/4493/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/106/builds/3411/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/23/builds/4762/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/62/builds/4467/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/111/builds/2474/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/48/builds/4473/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/37/builds/4473/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/104/builds/3358/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/113/builds/1819/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/580/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/580/steps/16/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/100/builds/2282/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/44/builds/4497/steps/18/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/44/builds/4497/steps/21/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/44/builds/4497/steps/24/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/64/builds/4466/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/45/builds/4480/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/53/builds/4476/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3349/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/47/builds/4473/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/47/builds/4473/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/42/builds/4457/steps/16/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/60/builds/4448/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/2458/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/74/builds/4432/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/63/builds/4434/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/107/builds/2439/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/52/builds/4430/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/108/builds/2416/steps/11/logs/stdio A previous build showed the same behavior for meta-aws: https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/542/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/542/steps/16/logs/stdio *** Bug 14594 has been marked as a duplicate of this bug. *** Could be a signature instability in Rust pkgs or it could just be a network instability. This looks like the crate fetcher problem I reported on the mailing list: https://lists.openembedded.org/g/openembedded-core/message/159725 Any setscene task for a recipe that inherits crate-fetch.bbclass will fail if the sstate package has to be fetched from the sstate mirror. See also this thread for a discussion of a proposed solution: https://lists.openembedded.org/g/openembedded-core/message/159828 This is only an issue on honister since master switched to a crate fetcher inside bitbake which resolved the issue. It did affect a number of targets in the 3.4.3 rc1 build: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3405 I logged onto one of the workers and got a bit more data from that failure. If I "git cherry-pick e222be06386698e24c664171bd6a1ea07bf28efa" then the warning seen becomes: WARNING: rust-cross-core2-32-musl-1.54.0-r0 do_deploy_source_date_epoch_setscene: ExpansionError('SRCPV', '${@crate_get_srcrev(d)}', FetchError('SRCREV was used yet no valid SCM was found in SRC_URI', None)) WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/x86_64-linux/rust-cross-core2-32-musl/1.54.0-r0/temp/log.do_deploy_source_date_epoch_setscene.3133099 WARNING: Setscene task (/home/pokybuild/yocto-worker/musl-qemux86/build/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead which gives a much better hint about what the issue is. To reproduce you need a setup with something like: SSTATE_MIRRORS += 'file://.* file:///srv/autobuilder/autobuilder.yocto.io/pub/sstate/PATH' SSTATE_DIR ?= '/srv/autobuilder/downloads.yoctoproject.org/sstate/3.4.3' then: rm /srv/autobuilder/downloads.yoctoproject.org/sstate/3.4.3/universal/7c/85/sstate\:rust-cross-core2-32-musl\:x86_64-poky-linux-musl\:1.54.0\:r0\:x86_64_i686\:7\:7c85203254b439497ffa94e556627b2c683b969f21fea11fd6a1524620bce947_deploy_source_date_epoch.tgz* bitbake rust-cross-core2-32-musl -c clean bitbake rust-cross-core2-32-musl -c deploy_source_date_epoch In sstate.bbclass, when fetching an sstate object we have: pstaging_fetch(sstatefetch, d): [...] localdata.setVar('SRCPV', d.getVar('SRCPV')) i.e. some code which expands SRCPV before it changes SRC_URI for the sstate tarball fetching. In crate-fetch.bbclass we have: def import_crate(d): import crate if not getattr(crate, 'imported', False): bb.fetch2.methods.append(crate.Crate()) crate.imported = True def crate_get_srcrev(d): import_crate(d) return bb.fetch2.get_srcrev(d) # Override SRCPV to make sure it imports the fetcher first SRCPV = "${@crate_get_srcrev(d)}" and so an "import crate" occurs when pstating_fetch() is called. That succeeds and all is well but the bb.fetch2.get_srcrev(d) call fails since there is no url in SRC_URI which supports srcrev(). Our options are: a) Ignore the issue and release. I suspect the TSC won't accept this. b) Backport adding the crate fetcher to bitbake. It is an invasive set of changes and I think too risky. c) Hack a try/except ignore into sstate.bbclass. This again is risky as it hits many code paths. d) Hack crate-fetch.bbclass. This may be possible. Discounting a-c above, I tested and found this works on the autobuilder where I could reproduce the issue: diff --git a/meta/classes/crate-fetch.bbclass b/meta/classes/crate-fetch.bbclass index a7fa22b2a0..04d76c0de8 100644 --- a/meta/classes/crate-fetch.bbclass +++ b/meta/classes/crate-fetch.bbclass @@ -22,6 +22,9 @@ crate_import_handler[eventmask] = "bb.event.RecipePreFinalise" def crate_get_srcrev(d): import_crate(d) + srcuri = d.getVar("SRC_URI") + if "crate://" not in srcuri and "git://" not in srcuri: + return "Invalid" return bb.fetch2.get_srcrev(d) # Override SRCPV to make sure it imports the fetcher first |