14680 2022-01-08 00:28:23 +0000 intermittent setscene tasks failures 2022-03-24 13:40:58 +0000 1 1 7 Build System, Metadata & Runtime OE-Core oe-core other 3.4 x86 Multiple RESOLVED FIXED AB-INT AB-INT-NET Medium+ normal 3.4.3 1 alexandre.belloni anuj.mittal alexandre.belloni anuj.mittal matt randy.macleod richard.purdie thomas.perrot --- --- No (bug/feature does not impact docs) oldest_to_newest 92264 0 alexandre.belloni 2022-01-08 00:28:23 +0000 A single build had many warning due to setscene tasks failing. Interestingly, they are almost all rust related packages: stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_package_qa_setscene.18794 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_package_qa_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/x86_64-linux/rust-cross-cortexa8hf-neon-glibc/1.57.0-r0/temp/log.do_populate_lic_setscene.18841 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/rust-cross_1.57.0.bb:do_populate_lic_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_package_write_deb_setscene.36368 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_package_write_deb_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_package_write_rpm_setscene.36412 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_package_write_rpm_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_package_write_ipk_setscene.36400 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_package_write_ipk_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_packagedata_setscene.8276 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_packagedata_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_populate_sysroot_setscene.8347 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_package_setscene.8617 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_package_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/libstd-rs/1.57.0-r0/temp/log.do_deploy_source_date_epoch_setscene.9076 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/libstd-rs_1.57.0.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/x86_64-linux/rust-cross-cortexa8hf-neon-glibc/1.57.0-r0/temp/log.do_populate_sysroot_setscene.9109 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/rust-cross_1.57.0.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/x86_64-linux/rust-cross-cortexa8hf-neon-glibc/1.57.0-r0/temp/log.do_deploy_source_date_epoch_setscene.9397 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/rust-cross_1.57.0.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/x86_64-linux/cargo-native/1.57.0-r0/temp/log.do_populate_sysroot_setscene.15496 stdio: WARNING: Setscene task (virtual:native:/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/cargo/cargo_1.57.0.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/x86_64-linux/rust-native/1.57.0-r0/temp/log.do_populate_sysroot_setscene.15831 stdio: WARNING: Setscene task (virtual:native:/home/pokybuild/yocto-worker/beaglebone/build/meta/recipes-devtools/rust/rust_1.57.0.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead The only exception is meta-aws where the aws packages failed: stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-checksums/0.1.11-r0/temp/log.do_deploy_source_date_epoch_setscene.19554 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-checksums/aws-checksums_0.1.11.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-crt-cpp/0.15.0-r0/temp/log.do_deploy_source_date_epoch_setscene.19555 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-crt-cpp/aws-crt-cpp_0.15.0.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-compression/0.2.14-r0/temp/log.do_deploy_source_date_epoch_setscene.19548 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-compression/aws-c-compression_0.2.14.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-iot/0.0.7-r0/temp/log.do_deploy_source_date_epoch_setscene.19544 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-cal/0.5.11-r0/temp/log.do_deploy_source_date_epoch_setscene.19550 stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-http/0.6.5-r0/temp/log.do_deploy_source_date_epoch_setscene.19549 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-cal/aws-c-cal_0.5.11.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-http/aws-c-http_0.6.5.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/temp/log.do_deploy_source_date_epoch_setscene.19547 stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-io/0.10.7-r0/temp/log.do_deploy_source_date_epoch_setscene.19551 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-io/aws-c-io_0.10.7.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-auth/aws-c-auth_0.6.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-s3/0.1.23-r0/temp/log.do_deploy_source_date_epoch_setscene.19552 stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-mqtt/0.7.6-r0/temp/log.do_deploy_source_date_epoch_setscene.19546 stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-event-stream/0.2.7-r0/temp/log.do_deploy_source_date_epoch_setscene.19545 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-s3/aws-c-s3_0.1.23.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-mqtt/aws-c-mqtt_0.7.6.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-c-event-stream/aws-c-event-stream_0.2.7.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead stdio: WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-iot-device-sdk-cpp-v2/1.14.1-r0/temp/log.do_deploy_source_date_epoch_setscene.19556 stdio: WARNING: Setscene task (/home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-sdk/aws-iot-device-sdk-cpp-v2/aws-iot-device-sdk-cpp-v2_1.14.1.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead 92265 1 alexandre.belloni 2022-01-08 00:33:32 +0000 https://autobuilder.yoctoproject.org/typhoon/#builders/97/builds/3794/steps/15/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/61/builds/4448/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/50/builds/4459/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/65/builds/4493/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/106/builds/3411/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/23/builds/4762/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/62/builds/4467/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/111/builds/2474/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/48/builds/4473/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/37/builds/4473/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/104/builds/3358/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/113/builds/1819/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/580/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/580/steps/16/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/100/builds/2282/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/44/builds/4497/steps/18/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/44/builds/4497/steps/21/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/44/builds/4497/steps/24/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/64/builds/4466/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/45/builds/4480/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/53/builds/4476/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/110/builds/3349/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/47/builds/4473/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/47/builds/4473/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/42/builds/4457/steps/16/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/60/builds/4448/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/102/builds/2458/steps/13/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/74/builds/4432/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/63/builds/4434/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/107/builds/2439/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/52/builds/4430/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/108/builds/2416/steps/11/logs/stdio 92268 2 alexandre.belloni 2022-01-08 00:39:52 +0000 A previous build showed the same behavior for meta-aws: https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/542/steps/12/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#builders/122/builds/542/steps/16/logs/stdio 92286 3 alexandre.belloni 2022-01-13 00:27:19 +0000 *** Bug 14594 has been marked as a duplicate of this bug. *** 92311 4 randy.macleod 2022-01-13 15:47:02 +0000 Could be a signature instability in Rust pkgs or it could just be a network instability. 92348 5 matt 2022-01-18 12:08:13 +0000 This looks like the crate fetcher problem I reported on the mailing list: https://lists.openembedded.org/g/openembedded-core/message/159725 Any setscene task for a recipe that inherits crate-fetch.bbclass will fail if the sstate package has to be fetched from the sstate mirror. See also this thread for a discussion of a proposed solution: https://lists.openembedded.org/g/openembedded-core/message/159828 92841 6 richard.purdie 2022-03-24 09:44:28 +0000 This is only an issue on honister since master switched to a crate fetcher inside bitbake which resolved the issue. It did affect a number of targets in the 3.4.3 rc1 build: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3405 I logged onto one of the workers and got a bit more data from that failure. If I "git cherry-pick e222be06386698e24c664171bd6a1ea07bf28efa" then the warning seen becomes: WARNING: rust-cross-core2-32-musl-1.54.0-r0 do_deploy_source_date_epoch_setscene: ExpansionError('SRCPV', '${@crate_get_srcrev(d)}', FetchError('SRCREV was used yet no valid SCM was found in SRC_URI', None)) WARNING: Logfile for failed setscene task is /home/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/x86_64-linux/rust-cross-core2-32-musl/1.54.0-r0/temp/log.do_deploy_source_date_epoch_setscene.3133099 WARNING: Setscene task (/home/pokybuild/yocto-worker/musl-qemux86/build/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead which gives a much better hint about what the issue is. To reproduce you need a setup with something like: SSTATE_MIRRORS += 'file://.* file:///srv/autobuilder/autobuilder.yocto.io/pub/sstate/PATH' SSTATE_DIR ?= '/srv/autobuilder/downloads.yoctoproject.org/sstate/3.4.3' then: rm /srv/autobuilder/downloads.yoctoproject.org/sstate/3.4.3/universal/7c/85/sstate\:rust-cross-core2-32-musl\:x86_64-poky-linux-musl\:1.54.0\:r0\:x86_64_i686\:7\:7c85203254b439497ffa94e556627b2c683b969f21fea11fd6a1524620bce947_deploy_source_date_epoch.tgz* bitbake rust-cross-core2-32-musl -c clean bitbake rust-cross-core2-32-musl -c deploy_source_date_epoch In sstate.bbclass, when fetching an sstate object we have: pstaging_fetch(sstatefetch, d): [...] localdata.setVar('SRCPV', d.getVar('SRCPV')) i.e. some code which expands SRCPV before it changes SRC_URI for the sstate tarball fetching. In crate-fetch.bbclass we have: def import_crate(d): import crate if not getattr(crate, 'imported', False): bb.fetch2.methods.append(crate.Crate()) crate.imported = True def crate_get_srcrev(d): import_crate(d) return bb.fetch2.get_srcrev(d) # Override SRCPV to make sure it imports the fetcher first SRCPV = "${@crate_get_srcrev(d)}" and so an "import crate" occurs when pstating_fetch() is called. That succeeds and all is well but the bb.fetch2.get_srcrev(d) call fails since there is no url in SRC_URI which supports srcrev(). Our options are: a) Ignore the issue and release. I suspect the TSC won't accept this. b) Backport adding the crate fetcher to bitbake. It is an invasive set of changes and I think too risky. c) Hack a try/except ignore into sstate.bbclass. This again is risky as it hits many code paths. d) Hack crate-fetch.bbclass. This may be possible. Discounting a-c above, I tested and found this works on the autobuilder where I could reproduce the issue: diff --git a/meta/classes/crate-fetch.bbclass b/meta/classes/crate-fetch.bbclass index a7fa22b2a0..04d76c0de8 100644 --- a/meta/classes/crate-fetch.bbclass +++ b/meta/classes/crate-fetch.bbclass @@ -22,6 +22,9 @@ crate_import_handler[eventmask] = "bb.event.RecipePreFinalise" def crate_get_srcrev(d): import_crate(d) + srcuri = d.getVar("SRC_URI") + if "crate://" not in srcuri and "git://" not in srcuri: + return "Invalid" return bb.fetch2.get_srcrev(d) # Override SRCPV to make sure it imports the fetcher first 92842 7 richard.purdie 2022-03-24 13:40:58 +0000 https://git.yoctoproject.org/poky/commit/?h=honister&id=d8cd104e66e7a616f6cf6be67d3b36a798be4c7b