| Summary: | AB-INT: sstatetests.SStateMirrors.test_cdn_mirror_qemu* failures | ||
|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Alexandre Belloni <alexandre.belloni> |
| Component: | oe-selftest | Assignee: | Alexander Kanavin <alex.kanavin> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | High | CC: | alex.kanavin, alexandre.belloni, mhalstead, randy.macleod |
| Version: | unspecified | ||
| Target Milestone: | 5.0 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Alexandre Belloni
2023-11-29 22:46:29 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6258/steps/23/logs/stdio rocky9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2314/steps/25/logs/stdio This one has: [Errno 32] Broken pipeTraceback (most recent call last): File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/ui/knotty.py", line 647, in main ret, error = server.runCommand(["ping"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/server/process.py", line 503, in runCommand self.connection.send(command) File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/server/process.py", line 868, in send self._send(obj) File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/server/process.py", line 847, in _send self.writer.send_bytes(obj) File "/home/pokybuild/yocto-worker/a-quick/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.11/multiprocessing/connection.py", line 199, in send_bytes self._send_bytes(m[offset:offset + size]) File "/home/pokybuild/yocto-worker/a-quick/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/home/pokybuild/yocto-worker/a-quick/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6240/steps/25/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2315/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6244/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2316/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6248/steps/23/logs/stdio Missing objects in the cache: lzip_1.23.bb:do_collect_spdx_deps tcl_8.6.13.bb:do_collect_spdx_deps go-binary-native_1.20.10.bb:do_collect_spdx_deps zip_3.0.bb:do_collect_spdx_deps https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6251/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2317/steps/25/logs/stdio *** Bug 15253 has been marked as a duplicate of this bug. *** https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6274/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2321/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6267/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6264/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6263/steps/25/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6262/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2319/steps/23/logs/stdio Michael, I found a couple of CDN oddities that seem to be coming from infra rather than bitbake/oe-core code or tests: 1. /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/2d/29/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst exists but https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst does not 2. wget https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst returns 504 gateway timeout (but not always!) while /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst exists. akanavin@ubuntu2204-arm-1:~$ wget https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst --2023-12-12 12:59:09-- https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst Resolving cdn.jsdelivr.net (cdn.jsdelivr.net)... 2a04:4e42::485, 2a04:4e42:200::485, 2a04:4e42:400::485, ... Connecting to cdn.jsdelivr.net (cdn.jsdelivr.net)|2a04:4e42::485|:443... connected. HTTP request sent, awaiting response... 504 Gateway Timeout Retrying. akanavin@ubuntu2204-arm-1:~$ wget https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst --2023-12-12 13:01:09-- https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst Resolving cdn.jsdelivr.net (cdn.jsdelivr.net)... 2a04:4e42:400::485, 2a04:4e42:200::485, 2a04:4e42:600::485, ... Connecting to cdn.jsdelivr.net (cdn.jsdelivr.net)|2a04:4e42:400::485|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 60192161 (57M) [application/octet-stream] Saving to: ‘sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst.2’ sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aar 100%[=========================================================================================================================================>] 57.40M 58.9MB/s in 1.0s The CDN didn't have sstate:go-cross-cortexa57:x86_64-poky-linux because the origin server wasn't serving it. https://origin.sstate.yoctoproject.org/all/universal/2d/29/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7 a339844f64b_populate_sysroot.tar.zst was a 404 as well. I checked the server and it had stale cache data for /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/2d/29/ After forcing a refresh the file was available. I've disabled all special caching. The NFS metadata cache lasts for 300 seconds. Let's see if this happens again. Thanks! On my side, I'm going to add a test that runs the same presence checks against local nfs mounted cache, so when the whole test set completes, it becomes possible to establish what isn't in the local cache to begin with, and what failed to propagate to cdn. Closing as the remaining issues are tracked in 15335 |