15945 2025-08-01 05:57:35 +0000 AB-INT: fetch/sstate/CDN - Missing objects in the cache: Bad file descriptor 2026-05-07 20:34:33 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified x86 Multiple RESOLVED FIXED AB-INT Medium normal 5.3 1 mathieu.dubois-briand unassigned alex.kanavin mathieu.dubois-briand paul randy.macleod richard.purdie yoann.congal --- --- No (bug/feature does not impact docs) oldest_to_newest 102537 0 mathieu.dubois-briand 2025-08-01 05:57:35 +0000 New ticket to track issues on new CDN. The error is a bit different now. AssertionError: 3 != 0 : Missing objects in the cache: file://58/73/sstate:libxmu:cortexa57-poky-linux:1.2.1:r0:cortexa57:14:58733b8346e5f8348a2028bdb92f6d75a5c7fa2295218a66a2aae29d57a3d827_deploy_source_date_epoch.tar.zst file://88/9b/sstate:xmodmap:cortexa57-poky-linux:1.0.11:r0:cortexa57:14:889b5f495eb44bda0637ec2862ecef308ee36cad271a386655bc2a259d7b6d15_deploy_source_date_epoch.tar.zst file://5f/7a/sstate:xrandr:cortexa57-poky-linux:1.5.3:r0:cortexa57:14:5f7a44a99a6dc0ccde2477825280734822e286b695415f1dd696e82b75ab9e29_package.tar.zst Fetcher diagnostics: DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/58/73/sstate%3Alibxmu%3Acortexa57-poky-linux%3A1.2.1%3Ar0%3Acortexa57%3A14%3A58733b8346e5f8348a2028bdb92f6d75a5c7fa2295218a66a2aae29d57a3d827_deploy_source_date_epoch.tar.zst: <urlopen error [Errno 9] Bad file descriptor> DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/88/9b/sstate%3Axmodmap%3Acortexa57-poky-linux%3A1.0.11%3Ar0%3Acortexa57%3A14%3A889b5f495eb44bda0637ec2862ecef308ee36cad271a386655bc2a259d7b6d15_deploy_source_date_epoch.tar.zst: <urlopen error [Errno 9] Bad file descriptor> DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/5f/7a/sstate%3Axrandr%3Acortexa57-poky-linux%3A1.5.3%3Ar0%3Acortexa57%3A14%3A5f7a44a99a6dc0ccde2477825280734822e286b695415f1dd696e82b75ab9e29_package.tar.zst: <urlopen error [Errno 9] Bad file descriptor> 102541 1 mathieu.dubois-briand 2025-08-01 09:52:00 +0000 a-full debian12-vk-7 master completed at 2025-07-31 04:22:33+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2117/steps/25/logs/stdio 102554 2 randy.macleod 2025-08-07 14:33:56 +0000 Let's see how often this happens and if we can spot a pattern. 102623 3 mathieu.dubois-briand 2025-08-19 08:34:19 +0000 a-full debian12-vk-4 master completed at 2025-08-17 12:14:45+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2217/steps/25/logs/stdio 102687 4 mathieu.dubois-briand 2025-09-04 07:03:07 +0000 a-full ubuntu2404-vk-1 mathieu/master-next-test completed at 2025-09-03 16:11:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2313/steps/25/logs/stdio 102713 5 mathieu.dubois-briand 2025-09-08 12:51:22 +0000 a-full debian12-vk-3 mathieu/master-next completed at 2025-09-05 23:21:05+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2327/steps/25/logs/stdio We also have a lot of "HTTP Error 404: Not Found " in there. a-full opensuse155-vk-2 mathieu/master-next completed at 2025-09-07 21:38:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2334/steps/27/logs/stdio 102730 6 mathieu.dubois-briand 2025-09-10 09:05:37 +0000 I created a new ticket for the 404 issue listed above: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15971. 102746 7 randy.macleod 2025-09-11 15:05:55 +0000 Were these caused by build failures? If so, close. 102754 8 mathieu.dubois-briand 2025-09-11 15:45:08 +0000 I had a look, and the fails from August and September 4th are not explained by failed builds. So I will keep this ticket open, sorry :( 102769 9 mathieu.dubois-briand 2025-09-12 06:43:57 +0000 a-full debian12-vk-5 mathieu/master-next completed at 2025-09-10 20:15:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2360/steps/25/logs/stdio 102811 10 mathieu.dubois-briand 2025-09-25 08:03:17 +0000 a-full ubuntu2204-vk-3 master completed at 2025-09-25 04:03:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2439/steps/25/logs/stdio 102835 11 mathieu.dubois-briand 2025-09-26 08:16:39 +0000 a-full debian11-vk-1 mathieu/master-next completed at 2025-09-25 19:41:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2441/steps/25/logs/stdio 102852 12 mathieu.dubois-briand 2025-09-30 08:46:22 +0000 a-full ubuntu2404-vk-1 mathieu/master-next completed at 2025-09-28 16:27:41+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2453/steps/26/logs/stdio 103849 13 antonin.godard 2026-01-13 13:28:48 +0000 a-full rocky9-vk-3 master completed at 2026-01-12 10:22:47+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3033/steps/31/logs/stdio 103888 14 paul 2026-01-14 17:04:01 +0000 We discussed this today, Alex pointed out that the `urlopen error [Errno 9] Bad file descriptor` message suggests a problem locally rather than with the network or CDN. Perhaps we're hitting a limit on the number of open file descriptors, perhaps we're leaking file descriptors. We should carry a patch to dump some debug information (e.g. what file descriptors do we have open?) when this error occurs. 103889 15 alex.kanavin 2026-01-14 17:10:36 +0000 I also need to note that the last report (2026-01-13) is not showing 'bad file descriptor' or similar local issue, it is '404 not found', e.g. missing objects on the CDN. So the last time local issues happened was end of september. I'd say we should put patient under observation for now, and focus on things that do happen more frequently, and more recently. 103890 16 alex.kanavin 2026-01-14 17:14:43 +0000 Oh, and the 2026-01-13 build basically did not happen: https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3033 which likely explains why there are lots of missing CDN cache objects. 103966 17 yoann.congal 2026-01-19 22:03:24 +0000 whinlatter a-full ubuntu2404-vk-3 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/29/builds/3089/steps/31/logs/stdio AssertionError: 1 != 0 : Missing objects in the cache: file://fe/e4/sstate:libpciaccess:x86-64-v3-poky-linux:0.18.1:r0:x86-64-v3:14:fee462c5a301d61e4f434443108525ed524be22789a9452eabe9ad47264bad53_populate_sysroot.tar.zst Fetcher diagnostics: DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/fe/e4/sstate%3Alibpciaccess%3Ax86-64-v3-poky-linux%3A0.18.1%3Ar0%3Ax86-64-v3%3A14%3Afee462c5a301d61e4f434443108525ed524be22789a9452eabe9ad47264bad53_populate_sysroot.tar.zst: <urlopen error [Errno 9] Bad file descriptor> Note that this is an otherwise successful build 104004 18 antonin.godard 2026-01-21 15:34:27 +0000 a-full ubuntu2404-vk-2 agodard/master-next completed at 2026-01-19 23:06:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3087/steps/31/logs/stdio a-full opensuse156-vk-1 agodard/master-next completed at 2026-01-21 01:08:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3098/steps/33/logs/stdio a-full debian12-vk-1 master completed at 2026-01-16 11:11:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3069/steps/31/logs/stdio a-full fedora40-vk-4 master completed at 2026-01-20 07:07:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3093/steps/31/logs/stdio a-full ubuntu2404-vk-1 master-next&master completed at 2026-01-20 23:47:31+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3096/steps/31/logs/stdio a-full debian12-vk-2 master-next&master completed at 2026-01-21 07:55:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3099/steps/31/logs/stdio 104009 19 paul 2026-01-21 17:07:28 +0000 The error here suggests we may have exceeded limits on the number of open file descriptors. Can someone add diagnostics to the relevant code path to print how many open file descriptors we have when the error occurs? We should be able to trap the error in Python code to do this. 104044 20 antonin.godard 2026-01-23 08:06:25 +0000 a-full debian12-vk-9 agodard/master-next completed at 2026-01-22 23:37:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3107/steps/31/logs/stdio a-full ubuntu2204-vk-1 master-next&master completed at 2026-01-22 08:19:04+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3104/steps/31/logs/stdio a-full debian12-vk-2 master-next&master completed at 2026-01-22 20:25:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3106/steps/31/logs/stdio a-full ubuntu2204-vk-4 master-next&master completed at 2026-01-23 01:17:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3108/steps/31/logs/stdio 104050 21 paul 2026-01-23 10:45:30 +0000 There are multiple failure modes seen in the logs: 1) Failure with "urlopen error [Errno 9] Bad file descriptor", last seen 2026-01-20. In these cases we usually see 1 and at most 3 objects missing in the sstate cache. 2) Failure with "urlopen error [Errno 32] Broken pipe", only seen 2025-09-28. In this case there were 174 objects missing in the sstate cache. This mode hasn't been seen in a while. 3) Recent failures with "HTTP Error 404: Not Found", between 2026-01-19 and 2026-01-23. In these cases we see 10s to 1000s of objects missing in the sstate cache. This is a similar failure mode to https://bugzilla.yoctoproject.org/show_bug.cgi?id=15971, but it affects many recipes instead of just one. I think the recent cases of (3) should be split out into a separate bug. 104051 22 alex.kanavin 2026-01-23 10:53:12 +0000 I would also note that 404 errors are only valid if the main build itself was a success. If it wasn't (e.g. a sea of failures, or skips), then the needed objects may not have been produced by it, and they won't be present in sstate. 104064 23 antonin.godard 2026-01-26 08:23:29 +0000 a-full fedora40-vk-3 master completed at 2026-01-24 04:22:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3119/steps/31/logs/stdio a-full debian11-vk-1 agodard/master-next completed at 2026-01-23 17:24:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3110/steps/31/logs/stdio a-full debian13-vk-2 agodard/master-next&agodard/master-next-bis completed at 2026-01-23 19:57:41+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3115/steps/31/logs/stdio a-full alma8-vk-2 master-next&master completed at 2026-01-25 20:17:56+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3121/steps/33/logs/stdio 104085 24 yoann.congal 2026-01-28 10:51:43 +0000 scarthgap a-full debian12-vk-5 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3133/steps/31/logs/stdio Log extract: DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/6d/e7/sstate%3Alibxrender%3Acortexa57-poky-linux%3A0.9.11%3Ar0%3Acortexa57%3A12%3A6de783aff168ddc1d6621b42f9fcbb2b62061634fcd4ebea5d6d9141c682ce04_package.tar.zst: <urlopen error [Errno 9] Bad file descriptor> 104087 25 mathieu.dubois-briand 2026-01-28 14:28:13 +0000 a-full alma9-vk-2 master&master-next completed at 2026-01-27 23:24:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3130/steps/31/logs/stdio a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-01-28 07:10:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3132/steps/31/logs/stdio 104095 26 mathieu.dubois-briand 2026-01-29 14:45:51 +0000 a-full debian12-vk-3 mathieu/master-next&mathieu/master-next-test completed at 2026-01-29 03:24:27+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3136/steps/31/logs/stdio a-full rocky9-vk-2 mathieu/master-next completed at 2026-01-29 03:06:57+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3138/steps/31/logs/stdio 104124 27 mathieu.dubois-briand 2026-01-30 09:08:17 +0000 a-full alma9-vk-2 mathieu/master-next completed at 2026-01-30 00:02:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3140/steps/31/logs/stdio a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-01-30 06:45:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3142/steps/31/logs/stdio 104131 28 mathieu.dubois-briand 2026-01-31 09:25:11 +0000 a-full debian12-vk-5 mathieu/master-next completed at 2026-01-30 23:01:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3144/steps/31/logs/stdio a-full fedora42-vk-1 mathieu/master-next completed at 2026-01-31 06:51:53+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3148/steps/31/logs/stdio 104147 29 mathieu.dubois-briand 2026-02-04 12:04:32 +0000 a-full rocky9-vk-2 mathieu/master-next completed at 2026-01-31 23:01:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3153/steps/31/logs/stdio a-full alma8-vk-1 mathieu/master-next completed at 2026-01-31 21:38:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3154/steps/33/logs/stdio a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-02-01 20:23:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3157/steps/31/logs/stdio a-full opensuse155-vk-2 mathieu/master-next completed at 2026-02-02 18:22:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3159/steps/33/logs/stdio a-full ubuntu2404-vk-1 mathieu/master-next completed at 2026-02-03 23:13:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3163/steps/31/logs/stdio 104204 30 mathieu.dubois-briand 2026-02-09 12:14:29 +0000 a-full rocky8-vk-1 mathieu/master-next completed at 2026-02-07 23:32:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3176/steps/33/logs/stdio a-full debian12-vk-9 master&master-next completed at 2026-02-06 01:38:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3170/steps/31/logs/stdio a-full ubuntu2504-vk-1 mathieu/master-next completed at 2026-02-06 05:07:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3171/steps/31/logs/stdio a-full alma9-vk-2 mathieu/master-next completed at 2026-02-05 17:43:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3169/steps/31/logs/stdio 104210 31 mathieu.dubois-briand 2026-02-10 10:19:51 +0000 a-full fedora40-vk-4 mathieu/master-next completed at 2026-02-09 17:35:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3184/steps/31/logs/stdio 104231 32 mathieu.dubois-briand 2026-02-11 15:45:09 +0000 a-full rocky8-vk-1 mathieu/master-next completed at 2026-02-11 00:28:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3188/steps/33/logs/stdio a-full debian12-vk-7 mathieu/master-next-test&mathieu/master-next completed at 2026-02-11 00:48:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3191/steps/31/logs/stdio 104236 33 mathieu.dubois-briand 2026-02-12 07:59:25 +0000 a-full debian12-vk-8 mathieu/master-next completed at 2026-02-11 16:01:09+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3194/steps/31/logs/stdio 104293 34 mathieu.dubois-briand 2026-02-13 12:39:40 +0000 a-full debian12-vk-9 mathieu/master-next completed at 2026-02-12 15:30:40+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3203/steps/31/logs/stdio 104317 35 mathieu.dubois-briand 2026-02-16 11:47:51 +0000 a-full debian13-vk-2 master completed at 2026-02-16 04:19:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3224/steps/31/logs/stdio 104346 36 mathieu.dubois-briand 2026-02-19 08:38:58 +0000 a-full ubuntu2404-vk-3 mathieu/master-next completed at 2026-02-18 23:43:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3240/steps/31/logs/stdio 104375 37 mathieu.dubois-briand 2026-02-23 09:55:05 +0000 a-full opensuse156-vk-1 mathieu/master-next completed at 2026-02-20 23:22:08+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3246/steps/33/logs/stdio Is this one really related? reproducible ubuntu2204-vk-1 master completed at 2026-02-20 03:36:34+00:00 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260220-lw128g40 https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260220-lw128g40/bitbake-reproducibleB-extended.log https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260220-lw128g40/packages https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3406/steps/13/logs/stdio 104573 38 yoann.congal 2026-03-10 13:41:10 +0000 scarthgap a-full debian12-vk-3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3374 DEBUG: checkstatus() urlopen failed for http://sstate.yoctoproject.org/all/27/55/sstate%3Aopenssl%3Acortexa57-poky-linux%3A3.5.5%3Ar0%3Acortexa57%3A12%3A2755d16c3d567f526f10aa8b78fe71b964a5c9567f0155f33996bb9fce4498e9_create_spdx.tar.zst: <urlopen error [Errno 9] Bad file descriptor> (URL downloads fine) 104579 39 mathieu.dubois-briand 2026-03-11 14:15:52 +0000 a-full debian12-vk-4 mathieu/master-next completed at 2026-03-11 11:30:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3382/steps/31/logs/stdio 104621 40 yoann.congal 2026-03-12 23:21:46 +0000 Potential fix sent by Richard: [PATCH] fetch/wget: Improve connection error handling https://lore.kernel.org/bitbake-devel/20260312223619.3735215-1-richard.purdie@linuxfoundation.org/T/#u 104737 41 randy.macleod 2026-03-19 15:07:14 +0000 Seems to have been fixed by: https://git.openembedded.org/bitbake/commit/?id=e8ace6e16c78ddbc2cfe985b88b66800d99282b3 105385 42 yoann.congal 2026-05-07 20:34:33 +0000 a-full ubuntu2204-vk-3 scarthgap completed at 2026-05-07 19:45:43+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3791 (In reply to Randy MacLeod from comment #41) > Seems to have been fixed by: > https://git.openembedded.org/bitbake/commit/?id=e8ace6e16c78ddbc2cfe985b88b66800d99282b3 I should backport this on scarthgap