| Summary: | AB-INT: SSL UNEXPECTED_EOF_WHILE_READING errors | ||
|---|---|---|---|
| Product: | [Infrastructure] AutoBuilder | Reporter: | Mathieu Dubois-Briand <mathieu.dubois-briand> |
| Component: | autobuilder | Assignee: | Michael Halstead <mhalstead> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | High | CC: | infras.ab.watcher, Infras.watcher, paul, randy.macleod, tim.orling, yoann.congal |
| Version: | unspecified | ||
| Target Milestone: | 6.2 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Or maybe it's only on yocto gits. I see no github error on today failures, maybe I got that wrong at some point. qemuarm64-armhost ubuntu2604-vk-arm1 mathieu/master-next completed at 2026-09-09 17:07:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/8/builds/4644/steps/19/logs/stdio qemux86-64 alma10-vk-1 master-next&master completed at 2026-09-09 15:30:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/4719/steps/17/logs/stdio qemuarm64 ubuntu2404-vk-2 mathieu/master-next completed at 2026-09-09 17:08:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/4598/steps/16/logs/stdio qemuriscv64 ubuntu2404-vk-1 mathieu/master-next completed at 2026-09-09 18:33:58+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/2319/steps/16/logs/stdio qemux86 stream9-vk-1 mathieu/master-next completed at 2026-09-09 18:58:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/4595/steps/17/logs/stdio qemuarm ubuntu2604-vk-3 master-next&master completed at 2026-09-09 20:33:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/4648/steps/17/logs/stdio qemuriscv64 opensuse160-vk-2 master-next&master completed at 2026-09-09 20:11:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/2320/steps/17/logs/stdio indexing debian13-vk-1 master completed at 2026-09-08 12:02:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/65766/steps/18/logs/stdio a-full rocky10-vk-1 mathieu/master-next completed at 2026-09-09 21:33:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/4705/steps/25/logs/stdio indexing rocky10-vk-1 master completed at 2026-09-09 11:16:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/65859/steps/18/logs/stdio oe-selftest-armhost alma10-vk-arm1 mathieu/master-next completed at 2026-09-09 20:02:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4855/steps/14/logs/stdio oe-selftest-armhost ubuntu2604-vk-arm1 master-next&master completed at 2026-09-09 21:37:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4856/steps/14/logs/stdio oe-selftest-debian ubuntu2604-vk-3 master-next&master completed at 2026-09-09 21:06:56+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4820/steps/14/logs/stdio oe-selftest-armhost stream10-vk-arm1 mathieu/master-next completed at 2026-09-08 21:43:35+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4845/steps/17/logs/stdio Note: I put some instances of this bug into https://bugzilla.yoctoproject.org/show_bug.cgi?id=16233 "AB-INT: 502/520 and other errors on yocto gits" (lumped under vague infra issues). TLS issues we're seeing with Cloudflare on YP infrastructure. I've removed Cloudflare from the path for git.yoctoproject.org and git.openembedded.org which appears to have mitigated the issue. I'm not marking this resolved yet since the issue is still unsolved and may continue to cause issues for users. qemuarm64 debian12-vk-4 master completed at 2026-09-10 20:29:00+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/4615/steps/17/logs/stdio qemux86 ubuntu2504-vk-1 master-next&master completed at 2026-09-10 10:04:14+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/4601/steps/17/logs/stdio oe-selftest-debian debian13-vk-1 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-10 16:32:20+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4828/steps/14/logs/stdio oe-selftest-debian ubuntu2604-vk-3 mathieu/master-next&mathieu/master-next-tests completed at 2026-09-10 19:23:59+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4829/steps/14/logs/stdio oe-selftest-armhost ubuntu2604-vk-arm1 mathieu/master-next completed at 2026-09-11 00:00:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4864/steps/14/logs/stdio oe-selftest-debian ubuntu2510-vk-1 mathieu/master-next completed at 2026-09-10 23:14:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4830/steps/14/logs/stdio oe-selftest-fedora opensuse160-vk-1 mathieu/master-next completed at 2026-09-10 23:16:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4636/steps/14/logs/stdio oe-selftest-fedora opensuse160-vk-1 master-next&master completed at 2026-09-10 16:01:17+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4635/steps/14/logs/stdio oe-selftest-fedora alma9-vk-2 wrynose completed at 2026-09-11 20:41:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/48/builds/4646 qemuarm64 debian13-vk-2 wrynose completed at 2026-09-11 19:45:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/36/builds/4628 oe-selftest-debian debian12-vk-2 wrynose completed at 2026-09-11 20:31:32+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/4839 oe-selftest-armhost ubuntu2604-vk-arm1 wrynose completed at 2026-09-11 21:03:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/23/builds/4874 qemuarm debian11-vk-3 wrynose completed at 2026-09-11 20:01:03+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/16/builds/4664 Can I get more info about the green builds linked in comment #7 https://bugzilla.yoctoproject.org/show_bug.cgi?id=16421#c7)? I'll see what I can do for the continuing layerindex issues. (In reply to Michael Halstead from comment #8) > Can I get more info about the green builds linked in comment #7 > https://bugzilla.yoctoproject.org/show_bug.cgi?id=16421#c7)? I linked the green retries and not the red builds, sorry. The red builds are: oe-selftest-fedora opensuse160-vk-1 wrynose completed at 2026-09-09 18:44:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/48/builds/4627 qemuarm64 opensuse156-vk-1 wrynose completed at 2026-09-09 17:43:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/36/builds/4599 oe-selftest-debian debian12-vk-8 wrynose completed at 2026-09-09 18:32:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/4818 oe-selftest-armhost stream10-vk-arm1 wrynose completed at 2026-09-09 19:31:16+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/23/builds/4854 qemuarm debian12-vk-7 wrynose completed at 2026-09-09 17:57:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/16/builds/4646 The is a logging enhancement patch in master next. oe-selftest-armhost debian13-vk-arm1 master&master-next completed at 2026-09-15 13:48:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4894/steps/14/logs/stdio oe-selftest-fedora opensuse160-vk-2 agodard/master-next completed at 2026-09-15 16:34:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4666/steps/14/logs/stdio oe-selftest-armhost debian13-vk-arm1 agodard/master-next&agodard/master-next-bis completed at 2026-09-15 20:53:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4896/steps/14/logs/stdio qemuarm64 rocky9-vk-2 master&master-next completed at 2026-09-15 11:53:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/4650/steps/18/logs/stdio qemuarmv5 opensuse160-vk-2 master&master-next completed at 2026-09-15 11:39:21+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/80/builds/4480/steps/17/logs/stdio qemuarmv5 opensuse160-vk-2 agodard/master-next&agodard/master-next-bis completed at 2026-09-15 16:59:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/80/builds/4482/steps/16/logs/stdio qemux86 rocky8-vk-1 master&master-next completed at 2026-09-15 22:30:50+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/4661/steps/18/logs/stdio indexing alma10-vk-1 master completed at 2026-09-15 11:04:29+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/66422/steps/18/logs/stdio indexing opensuse160-vk-2 master completed at 2026-09-15 12:31:42+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/66428/steps/18/logs/stdio indexing opensuse160-vk-1 master completed at 2026-09-15 14:07:07+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/66433/steps/18/logs/stdio indexing ubuntu2604-vk-4 master completed at 2026-09-15 14:47:06+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/66437/steps/18/logs/stdio oe-selftest-fedora fedora44-vk-2 master&master-next completed at 2026-09-20 11:28:10+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4702/steps/14/logs/stdio indexing fedora44-vk-3 master completed at 2026-09-22 11:31:58+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/104/builds/67089/steps/18/logs/stdio oe-selftest-fedora fedora45-vk-1 master-next&master completed at 2026-09-22 19:27:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4716/steps/14/logs/stdio Michael was able to reproduce the issue, there was a 15 second gap and he's trying to work out the cause and fix. We haven't seen this in a while. Re-open if it happens again. |
We have seen various OpenSSL errors since the end of last week (Saturday IIRC). They happen both on layerindex, our own gits and github. I was first blaming that on the usual infrastructure issues because of scrappers, but there might be something else here. Actual errors tends to vary a bit. Here is one in the indexing job: + /srv/pokybuild/yocto-worker/indexing/yocto-autobuilder-helper/scripts/layer-parser.py Traceback (most recent call last): ... File "/usr/lib/python3.13/ssl.py", line 1372, in do_handshake self._sslobj.do_handshake() ~~~~~~~~~~~~~~~~~~~~~~~~~^^ ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1029) ... Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/indexing/yocto-autobuilder-helper/scripts/layer-parser.py", line 16, in <module> with urllib.request.urlopen(LAYER_API_URL) as response: ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ ... File "/usr/lib/python3.13/urllib/request.py", line 1322, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1029)> And another one during testimage/testsdk: oeqa.utils.subprocesstweak.OETestCalledProcessError: Command '. /srv/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/testsdkext/environment-setup-cortexa57-poky-linux > /dev/null; devtool add kernel-module-hello-world https://git.yoctoproject.org/kernel-module-hello-world;' returned non-zero exit status 1. ... bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command ['git', '-c', 'gc.autoDetach=false', '-c', 'core.pager=cat', '-c', 'safe.bareRepository=all', '-c', 'clone.defaultRemoteName=origin', 'ls-remote', 'https://git.yoctoproject.org/kernel-module-hello-world'] failed with exit code 128, output: fatal: unable to access 'https://git.yoctoproject.org/kernel-module-hello-world/': TLS connect error: error:0A000126:SSL routines::unexpected eof while reading