New ticket to track issues on new CDN. The error is a bit different now. AssertionError: 3 != 0 : Missing objects in the cache: file://58/73/sstate:libxmu:cortexa57-poky-linux:1.2.1:r0:cortexa57:14:58733b8346e5f8348a2028bdb92f6d75a5c7fa2295218a66a2aae29d57a3d827_deploy_source_date_epoch.tar.zst file://88/9b/sstate:xmodmap:cortexa57-poky-linux:1.0.11:r0:cortexa57:14:889b5f495eb44bda0637ec2862ecef308ee36cad271a386655bc2a259d7b6d15_deploy_source_date_epoch.tar.zst file://5f/7a/sstate:xrandr:cortexa57-poky-linux:1.5.3:r0:cortexa57:14:5f7a44a99a6dc0ccde2477825280734822e286b695415f1dd696e82b75ab9e29_package.tar.zst Fetcher diagnostics: DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/58/73/sstate%3Alibxmu%3Acortexa57-poky-linux%3A1.2.1%3Ar0%3Acortexa57%3A14%3A58733b8346e5f8348a2028bdb92f6d75a5c7fa2295218a66a2aae29d57a3d827_deploy_source_date_epoch.tar.zst: <urlopen error [Errno 9] Bad file descriptor> DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/88/9b/sstate%3Axmodmap%3Acortexa57-poky-linux%3A1.0.11%3Ar0%3Acortexa57%3A14%3A889b5f495eb44bda0637ec2862ecef308ee36cad271a386655bc2a259d7b6d15_deploy_source_date_epoch.tar.zst: <urlopen error [Errno 9] Bad file descriptor> DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/5f/7a/sstate%3Axrandr%3Acortexa57-poky-linux%3A1.5.3%3Ar0%3Acortexa57%3A14%3A5f7a44a99a6dc0ccde2477825280734822e286b695415f1dd696e82b75ab9e29_package.tar.zst: <urlopen error [Errno 9] Bad file descriptor>
a-full debian12-vk-7 master completed at 2025-07-31 04:22:33+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2117/steps/25/logs/stdio
Let's see how often this happens and if we can spot a pattern.
a-full debian12-vk-4 master completed at 2025-08-17 12:14:45+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2217/steps/25/logs/stdio
a-full ubuntu2404-vk-1 mathieu/master-next-test completed at 2025-09-03 16:11:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2313/steps/25/logs/stdio
a-full debian12-vk-3 mathieu/master-next completed at 2025-09-05 23:21:05+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2327/steps/25/logs/stdio We also have a lot of "HTTP Error 404: Not Found " in there. a-full opensuse155-vk-2 mathieu/master-next completed at 2025-09-07 21:38:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2334/steps/27/logs/stdio
I created a new ticket for the 404 issue listed above: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15971.
Were these caused by build failures? If so, close.
I had a look, and the fails from August and September 4th are not explained by failed builds. So I will keep this ticket open, sorry :(
a-full debian12-vk-5 mathieu/master-next completed at 2025-09-10 20:15:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2360/steps/25/logs/stdio
a-full ubuntu2204-vk-3 master completed at 2025-09-25 04:03:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2439/steps/25/logs/stdio
a-full debian11-vk-1 mathieu/master-next completed at 2025-09-25 19:41:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2441/steps/25/logs/stdio
a-full ubuntu2404-vk-1 mathieu/master-next completed at 2025-09-28 16:27:41+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2453/steps/26/logs/stdio
a-full rocky9-vk-3 master completed at 2026-01-12 10:22:47+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3033/steps/31/logs/stdio
We discussed this today, Alex pointed out that the `urlopen error [Errno 9] Bad file descriptor` message suggests a problem locally rather than with the network or CDN. Perhaps we're hitting a limit on the number of open file descriptors, perhaps we're leaking file descriptors. We should carry a patch to dump some debug information (e.g. what file descriptors do we have open?) when this error occurs.
I also need to note that the last report (2026-01-13) is not showing 'bad file descriptor' or similar local issue, it is '404 not found', e.g. missing objects on the CDN. So the last time local issues happened was end of september. I'd say we should put patient under observation for now, and focus on things that do happen more frequently, and more recently.
Oh, and the 2026-01-13 build basically did not happen: https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3033 which likely explains why there are lots of missing CDN cache objects.
whinlatter a-full ubuntu2404-vk-3 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/29/builds/3089/steps/31/logs/stdio AssertionError: 1 != 0 : Missing objects in the cache: file://fe/e4/sstate:libpciaccess:x86-64-v3-poky-linux:0.18.1:r0:x86-64-v3:14:fee462c5a301d61e4f434443108525ed524be22789a9452eabe9ad47264bad53_populate_sysroot.tar.zst Fetcher diagnostics: DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/fe/e4/sstate%3Alibpciaccess%3Ax86-64-v3-poky-linux%3A0.18.1%3Ar0%3Ax86-64-v3%3A14%3Afee462c5a301d61e4f434443108525ed524be22789a9452eabe9ad47264bad53_populate_sysroot.tar.zst: <urlopen error [Errno 9] Bad file descriptor> Note that this is an otherwise successful build
a-full ubuntu2404-vk-2 agodard/master-next completed at 2026-01-19 23:06:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3087/steps/31/logs/stdio a-full opensuse156-vk-1 agodard/master-next completed at 2026-01-21 01:08:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3098/steps/33/logs/stdio a-full debian12-vk-1 master completed at 2026-01-16 11:11:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3069/steps/31/logs/stdio a-full fedora40-vk-4 master completed at 2026-01-20 07:07:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3093/steps/31/logs/stdio a-full ubuntu2404-vk-1 master-next&master completed at 2026-01-20 23:47:31+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3096/steps/31/logs/stdio a-full debian12-vk-2 master-next&master completed at 2026-01-21 07:55:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3099/steps/31/logs/stdio
The error here suggests we may have exceeded limits on the number of open file descriptors. Can someone add diagnostics to the relevant code path to print how many open file descriptors we have when the error occurs? We should be able to trap the error in Python code to do this.
a-full debian12-vk-9 agodard/master-next completed at 2026-01-22 23:37:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3107/steps/31/logs/stdio a-full ubuntu2204-vk-1 master-next&master completed at 2026-01-22 08:19:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3104/steps/31/logs/stdio a-full debian12-vk-2 master-next&master completed at 2026-01-22 20:25:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3106/steps/31/logs/stdio a-full ubuntu2204-vk-4 master-next&master completed at 2026-01-23 01:17:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3108/steps/31/logs/stdio
There are multiple failure modes seen in the logs: 1) Failure with "urlopen error [Errno 9] Bad file descriptor", last seen 2026-01-20. In these cases we usually see 1 and at most 3 objects missing in the sstate cache. 2) Failure with "urlopen error [Errno 32] Broken pipe", only seen 2025-09-28. In this case there were 174 objects missing in the sstate cache. This mode hasn't been seen in a while. 3) Recent failures with "HTTP Error 404: Not Found", between 2026-01-19 and 2026-01-23. In these cases we see 10s to 1000s of objects missing in the sstate cache. This is a similar failure mode to https://bugzilla.yoctoproject.org/show_bug.cgi?id=15971, but it affects many recipes instead of just one. I think the recent cases of (3) should be split out into a separate bug.
I would also note that 404 errors are only valid if the main build itself was a success. If it wasn't (e.g. a sea of failures, or skips), then the needed objects may not have been produced by it, and they won't be present in sstate.
a-full fedora40-vk-3 master completed at 2026-01-24 04:22:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3119/steps/31/logs/stdio a-full debian11-vk-1 agodard/master-next completed at 2026-01-23 17:24:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3110/steps/31/logs/stdio a-full debian13-vk-2 agodard/master-next&agodard/master-next-bis completed at 2026-01-23 19:57:41+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3115/steps/31/logs/stdio a-full alma8-vk-2 master-next&master completed at 2026-01-25 20:17:56+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3121/steps/33/logs/stdio
scarthgap a-full debian12-vk-5 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3133/steps/31/logs/stdio Log extract: DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/6d/e7/sstate%3Alibxrender%3Acortexa57-poky-linux%3A0.9.11%3Ar0%3Acortexa57%3A12%3A6de783aff168ddc1d6621b42f9fcbb2b62061634fcd4ebea5d6d9141c682ce04_package.tar.zst: <urlopen error [Errno 9] Bad file descriptor>
a-full alma9-vk-2 master&master-next completed at 2026-01-27 23:24:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3130/steps/31/logs/stdio a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-01-28 07:10:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3132/steps/31/logs/stdio
a-full debian12-vk-3 mathieu/master-next&mathieu/master-next-test completed at 2026-01-29 03:24:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3136/steps/31/logs/stdio a-full rocky9-vk-2 mathieu/master-next completed at 2026-01-29 03:06:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3138/steps/31/logs/stdio
a-full alma9-vk-2 mathieu/master-next completed at 2026-01-30 00:02:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3140/steps/31/logs/stdio a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-01-30 06:45:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3142/steps/31/logs/stdio
a-full debian12-vk-5 mathieu/master-next completed at 2026-01-30 23:01:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3144/steps/31/logs/stdio a-full fedora42-vk-1 mathieu/master-next completed at 2026-01-31 06:51:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3148/steps/31/logs/stdio
a-full rocky9-vk-2 mathieu/master-next completed at 2026-01-31 23:01:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3153/steps/31/logs/stdio a-full alma8-vk-1 mathieu/master-next completed at 2026-01-31 21:38:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3154/steps/33/logs/stdio a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-02-01 20:23:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3157/steps/31/logs/stdio a-full opensuse155-vk-2 mathieu/master-next completed at 2026-02-02 18:22:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3159/steps/33/logs/stdio a-full ubuntu2404-vk-1 mathieu/master-next completed at 2026-02-03 23:13:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3163/steps/31/logs/stdio
a-full rocky8-vk-1 mathieu/master-next completed at 2026-02-07 23:32:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3176/steps/33/logs/stdio a-full debian12-vk-9 master&master-next completed at 2026-02-06 01:38:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3170/steps/31/logs/stdio a-full ubuntu2504-vk-1 mathieu/master-next completed at 2026-02-06 05:07:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3171/steps/31/logs/stdio a-full alma9-vk-2 mathieu/master-next completed at 2026-02-05 17:43:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3169/steps/31/logs/stdio
a-full fedora40-vk-4 mathieu/master-next completed at 2026-02-09 17:35:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3184/steps/31/logs/stdio
a-full rocky8-vk-1 mathieu/master-next completed at 2026-02-11 00:28:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3188/steps/33/logs/stdio a-full debian12-vk-7 mathieu/master-next-test&mathieu/master-next completed at 2026-02-11 00:48:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3191/steps/31/logs/stdio
a-full debian12-vk-8 mathieu/master-next completed at 2026-02-11 16:01:09+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3194/steps/31/logs/stdio
a-full debian12-vk-9 mathieu/master-next completed at 2026-02-12 15:30:40+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3203/steps/31/logs/stdio
a-full debian13-vk-2 master completed at 2026-02-16 04:19:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3224/steps/31/logs/stdio
a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-02-18 23:43:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3240/steps/31/logs/stdio
a-full opensuse156-vk-1 mathieu/master-next completed at 2026-02-20 23:22:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3246/steps/33/logs/stdio Is this one really related? reproducible ubuntu2204-vk-1 master completed at 2026-02-20 03:36:34+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260220-lw128g40 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260220-lw128g40/bitbake-reproducibleB-extended.log https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260220-lw128g40/packages https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3406/steps/13/logs/stdio
scarthgap a-full debian12-vk-3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3374 DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/27/55/sstate%3Aopenssl%3Acortexa57-poky-linux%3A3.5.5%3Ar0%3Acortexa57%3A12%3A2755d16c3d567f526f10aa8b78fe71b964a5c9567f0155f33996bb9fce4498e9_create_spdx.tar.zst: <urlopen error [Errno 9] Bad file descriptor> (URL downloads fine)
a-full debian12-vk-4 mathieu/master-next completed at 2026-03-11 11:30:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3382/steps/31/logs/stdio
Potential fix sent by Richard: [PATCH] fetch/wget: Improve connection error handling https://lore.kernel.org/bitbake-devel/20260312223619.3735215-1-richard.purdie@linuxfoundation.org/T/#u
Seems to have been fixed by: https://git.openembedded.org/bitbake/commit/?id=e8ace6e16c78ddbc2cfe985b88b66800d99282b3
a-full ubuntu2204-vk-3 scarthgap completed at 2026-05-07 19:45:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3791 (In reply to Randy MacLeod from comment #41) > Seems to have been fixed by: > https://git.openembedded.org/bitbake/commit/?id=e8ace6e16c78ddbc2cfe985b88b66800d99282b3 I should backport this on scarthgap