15303 2023-11-29 22:46:29 +0000 AB-INT: sstatetests.SStateMirrors.test_cdn_mirror_qemu* failures 2024-01-05 15:16:23 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified x86 Multiple RESOLVED FIXED AB-INT High normal 5.0 M2 1 alexandre.belloni alex.kanavin alex.kanavin alexandre.belloni mhalstead randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 97363 0 alexandre.belloni 2023-11-29 22:46:29 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2318/steps/23/logs/stdio a-quick fedora38-ty-6 97366 1 alexandre.belloni 2023-11-29 23:05:39 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6258/steps/23/logs/stdio rocky9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2314/steps/25/logs/stdio This one has: [Errno 32] Broken pipeTraceback (most recent call last): File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/ui/knotty.py", line 647, in main ret, error = server.runCommand(["ping"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/server/process.py", line 503, in runCommand self.connection.send(command) File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/server/process.py", line 868, in send self._send(obj) File "/home/pokybuild/yocto-worker/a-quick/build/bitbake/lib/bb/server/process.py", line 847, in _send self.writer.send_bytes(obj) File "/home/pokybuild/yocto-worker/a-quick/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.11/multiprocessing/connection.py", line 199, in send_bytes self._send_bytes(m[offset:offset + size]) File "/home/pokybuild/yocto-worker/a-quick/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/home/pokybuild/yocto-worker/a-quick/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe 97367 2 alexandre.belloni 2023-11-29 23:10:05 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6240/steps/25/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2315/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6244/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2316/steps/23/logs/stdio 97368 3 alexandre.belloni 2023-11-29 23:10:17 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6248/steps/23/logs/stdio Missing objects in the cache: lzip_1.23.bb:do_collect_spdx_deps tcl_8.6.13.bb:do_collect_spdx_deps go-binary-native_1.20.10.bb:do_collect_spdx_deps zip_3.0.bb:do_collect_spdx_deps 97369 4 alexandre.belloni 2023-11-29 23:11:58 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6251/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2317/steps/25/logs/stdio 97372 5 alexandre.belloni 2023-11-29 23:21:08 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6229/steps/25/logs/stdio 97384 6 alex.kanavin 2023-11-30 16:18:57 +0000 *** Bug 15253 has been marked as a duplicate of this bug. *** 97409 7 alexandre.belloni 2023-12-04 11:15:44 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6274/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2321/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6267/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6264/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6263/steps/25/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6262/steps/23/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2319/steps/23/logs/stdio 97411 8 alexandre.belloni 2023-12-04 11:40:20 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2322/steps/25/logs/stdio 97429 9 alexandre.belloni 2023-12-06 11:08:29 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6259/steps/23/logs/stdio 97430 10 alexandre.belloni 2023-12-06 11:10:14 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6291/steps/23/logs/stdio 97544 11 alex.kanavin 2023-12-12 13:00:41 +0000 Michael, I found a couple of CDN oddities that seem to be coming from infra rather than bitbake/oe-core code or tests: 1. /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/2d/29/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst exists but https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst does not 2. wget https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst returns 504 gateway timeout (but not always!) while /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst exists. 97545 12 alex.kanavin 2023-12-12 13:03:28 +0000 akanavin@ubuntu2204-arm-1:~$ wget https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst --2023-12-12 12:59:09-- https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst Resolving cdn.jsdelivr.net (cdn.jsdelivr.net)... 2a04:4e42::485, 2a04:4e42:200::485, 2a04:4e42:400::485, ... Connecting to cdn.jsdelivr.net (cdn.jsdelivr.net)|2a04:4e42::485|:443... connected. HTTP request sent, awaiting response... 504 Gateway Timeout Retrying. akanavin@ubuntu2204-arm-1:~$ wget https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst --2023-12-12 13:01:09-- https://cdn.jsdelivr.net/yocto/sstate/all/universal/2d/29/sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst Resolving cdn.jsdelivr.net (cdn.jsdelivr.net)... 2a04:4e42:400::485, 2a04:4e42:200::485, 2a04:4e42:600::485, ... Connecting to cdn.jsdelivr.net (cdn.jsdelivr.net)|2a04:4e42:400::485|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 60192161 (57M) [application/octet-stream] Saving to: ‘sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aarch64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7a339844f64b_populate_sysroot.tar.zst.2’ sstate:go-cross-cortexa57:aarch64-poky-linux:1.20.12:r0:aar 100%[=========================================================================================================================================>] 57.40M 58.9MB/s in 1.0s 97557 13 mhalstead 2023-12-13 19:04:02 +0000 The CDN didn't have sstate:go-cross-cortexa57:x86_64-poky-linux because the origin server wasn't serving it. https://origin.sstate.yoctoproject.org/all/universal/2d/29/sstate:go-cross-cortexa57:x86_64-poky-linux:1.20.12:r0:x86_64:11:2d291879552b44a099c36e9e85358f703a8b8d8c2c0c2b876a0c7 a339844f64b_populate_sysroot.tar.zst was a 404 as well. I checked the server and it had stale cache data for /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/2d/29/ After forcing a refresh the file was available. I've disabled all special caching. The NFS metadata cache lasts for 300 seconds. Let's see if this happens again. 97558 14 alex.kanavin 2023-12-13 19:10:13 +0000 Thanks! On my side, I'm going to add a test that runs the same presence checks against local nfs mounted cache, so when the whole test set completes, it becomes possible to establish what isn't in the local cache to begin with, and what failed to propagate to cdn. 97743 15 alexandre.belloni 2024-01-05 15:16:23 +0000 Closing as the remaining issues are tracked in 15335